From nobody Mon Feb 26 00:10:16 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TjgxW57VHz5Bdvb for ; Mon, 26 Feb 2024 00:10:39 +0000 (UTC) (envelope-from void@f-m.fm) Received: from fout3-smtp.messagingengine.com (fout3-smtp.messagingengine.com [103.168.172.146]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TjgxV5cbWz4pPG for ; Mon, 26 Feb 2024 00:10:38 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm1 header.b=Xa9v+EHl; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=U9IN1g23; dmarc=pass (policy=none) header.from=f-m.fm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 103.168.172.146 as permitted sender) smtp.mailfrom=void@f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.nyi.internal (Postfix) with ESMTP id 0970A13800C5 for ; Sun, 25 Feb 2024 19:10:38 -0500 (EST) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Sun, 25 Feb 2024 19:10:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1708906238; x=1708992638; bh=vvEUXyaEpz bk+u53BQ3qxveMQeE89ncMyxTM3y0Es2Q=; b=Xa9v+EHlsGh6rr1UjR/PFbLubb Jpw/BdwIj0RSa3+Z2YUQqCImILgqKIYUrKAjIgLcdtQ928QBAudHlPvPJ5c/YKCM e7jbEDeMOika0l8mjcwPwXEvlWzmimULrZGpuSW3/8uTvuQFMopvTypDf4yCsvMm QSsVYv5QKIOSIVE3YgV/u4T86h4gcbzH362UPbZJzuQSV2STr82JSRzpB0ONYUFU 979pei3e/76TFFfZHJlfhlOWL46WJFBrdFWRi97JtbXqXVcUMOwvI6Myj7T9GC8f ypsjGBzco26JpYvAEYUQ9njtvXig8ED0t/G7o2gPPCTtnUBtaQaWYYiOkyfQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1708906238; x=1708992638; bh=vvEUXyaEpzbk+u53BQ3qxveMQeE8 9ncMyxTM3y0Es2Q=; b=U9IN1g23r1waqWoI9NVL4Evc7fEF+0O0U9OiCnzUkuw9 WGgUruRe3rbuhh9+q9XwCFb/CIgdVFgA3pMTkQ5C94bR9YUA/XqCI5AklppgVhIJ GwsWFsOQu/DwibxKUswmdE75Ok/cSHsjHr4BIOzWmqznbPgIpZaTngSQoQSpB5a0 kUxAbKZzKwu+1HooCcuHjv8NYmheLlB9qICiMqWoKMqsE8PJS4Gxu+nLU/iD+BkQ zHKjff7pVdjkX0BihrCqR4bwcaVF+cRUCle2Kp0VEnx4VDML2US4uqiSkourae5P xY9p71HzI88Y141WPvsyqBSFLFDH7aanqOdtpzxfTQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgedugddulecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfggfkjghffffhvffutgesthdtre dtreertdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepieetvdeuhedthedtvdfhuefhveehvdeiledvieffheevleehgeefudelje dukedvnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id BF4EA2A2008B; Sun, 25 Feb 2024 19:10:37 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-153-g7e3bb84806-fm-20240215.007-g7e3bb848 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Message-Id: <5a581a59-94fc-4476-97c4-4afac65c27bf@app.fastmail.com> In-Reply-To: <478521708877670@z7oys7c4urhdnodv.sas.yp-c.yandex.net> References: <5813111708783942@wckjnuw7tehz7zgt.sas.yp-c.yandex.net> <3c8667a2-770c-4c35-9b33-572f43d9f1ea@app.fastmail.com> <4609701708855679@vpfns56j3qv6ec2k.vla.yp-c.yandex.net> <511954d9-74b4-45c3-b16e-342ad29751f7@app.fastmail.com> <478521708877670@z7oys7c4urhdnodv.sas.yp-c.yandex.net> Date: Mon, 26 Feb 2024 00:10:16 +0000 From: void To: stable@freebsd.org Subject: Re: USB CD drive does not work with 14-Stable Content-Type: text/plain X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.09 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm1,messagingengine.com:s=fm1]; R_SPF_ALLOW(-0.20)[+ip4:103.168.172.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[103.168.172.146:from]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; FREEMAIL_FROM(0.00)[f-m.fm]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+] X-Rspamd-Queue-Id: 4TjgxV5cbWz4pPG On Sun, 25 Feb 2024, at 16:14, S.N.Grigoriev wrote: >> Another thing maybe worthwhile trying is to just have that one thing >> plugged into the usb subsystem. Beforehand, tail -f /var/log/messages >> in an xterm, unplug everything usb (2&3), plug in the cd drive, >> observe messages. > > I've done it. The same result. What is the output of lsusb -v ? -- From nobody Mon Feb 26 10:21:04 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TjxTy70WDz5CqS0; Mon, 26 Feb 2024 10:21:10 +0000 (UTC) (envelope-from serguey-grigoriev@yandex.ru) Received: from forward502a.mail.yandex.net (forward502a.mail.yandex.net [IPv6:2a02:6b8:c0e:500:1:45:d181:d502]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TjxTx4qb4z4wgh; Mon, 26 Feb 2024 10:21:09 +0000 (UTC) (envelope-from serguey-grigoriev@yandex.ru) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yandex.ru header.s=mail header.b=pgVFKTPJ; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (mx1.freebsd.org: domain of serguey-grigoriev@yandex.ru designates 2a02:6b8:c0e:500:1:45:d181:d502 as permitted sender) smtp.mailfrom=serguey-grigoriev@yandex.ru Received: from mail-nwsmtp-mxback-production-main-90.iva.yp-c.yandex.net (mail-nwsmtp-mxback-production-main-90.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:2da1:0:640:d211:0]) by forward502a.mail.yandex.net (Yandex) with ESMTPS id 9ED1A60BDD; Mon, 26 Feb 2024 13:21:05 +0300 (MSK) Received: from mail.yandex.ru (2a02:6b8:c0c:3b18:0:640:873a:0 [2a02:6b8:c0c:3b18:0:640:873a:0]) by mail-nwsmtp-mxback-production-main-90.iva.yp-c.yandex.net (mxback/Yandex) with HTTP id 5LUH4D0M6Gk0-fYZGIFpI; Mon, 26 Feb 2024 13:21:05 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1708942865; bh=klzeLPyysPIhk2x5BxLj6xPisi2UnUtW05iCIIQbQ3k=; h=Message-Id:References:Date:Cc:Subject:In-Reply-To:To:From; b=pgVFKTPJNpsBeY9G5sp8z06cZlWiofflSnfKJB+ZVXbxbDMOsxOG6Djz0knZnkC7o f+TjI+qzg1NDhbAiP2FIkCp8rHjIAnRfk4nomq7Q9arG7gP73M52YOXGxqBMy4IMyF 98zAabkWiyDGdWMjQrIw6B34EduGh+MJaNfGiwJw= Received: by omv3tj5zwvd3plqc.iva.yp-c.yandex.net with HTTP; Mon, 26 Feb 2024 13:21:04 +0300 From: S.N.Grigoriev To: FreeBSD Stable Cc: stable@freebsd.org In-Reply-To: <5a581a59-94fc-4476-97c4-4afac65c27bf@app.fastmail.com> References: <5813111708783942@wckjnuw7tehz7zgt.sas.yp-c.yandex.net> <3c8667a2-770c-4c35-9b33-572f43d9f1ea@app.fastmail.com> <4609701708855679@vpfns56j3qv6ec2k.vla.yp-c.yandex.net> <511954d9-74b4-45c3-b16e-342ad29751f7@app.fastmail.com> <478521708877670@z7oys7c4urhdnodv.sas.yp-c.yandex.net> <5a581a59-94fc-4476-97c4-4afac65c27bf@app.fastmail.com> Subject: Re: USB CD drive does not work with 14-Stable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Mon, 26 Feb 2024 13:21:04 +0300 Message-Id: <4874751708942864@omv3tj5zwvd3plqc.iva.yp-c.yandex.net> Content-Type: multipart/mixed; boundary="----==--bound.487476.omv3tj5zwvd3plqc.iva.yp-c.yandex.net" X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.20 / 15.00]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yandex.ru,none]; NEURAL_HAM_SHORT(-0.20)[-0.205]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:c00::/40]; R_DKIM_ALLOW(-0.20)[yandex.ru:s=mail]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[2a02:6b8:c0e:500:1:45:d181:d502:from]; MIME_BASE64_TEXT(0.10)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[yandex.ru]; MIME_TRACE(0.00)[0:+,1:+,2:+]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org,stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; HAS_ATTACHMENT(0.00)[]; DKIM_TRACE(0.00)[yandex.ru:+]; FREEMAIL_ENVFROM(0.00)[yandex.ru]; DWL_DNSWL_NONE(0.00)[yandex.ru:dkim] X-Rspamd-Queue-Id: 4TjxTx4qb4z4wgh ------==--bound.487476.omv3tj5zwvd3plqc.iva.yp-c.yandex.net Content-Transfer-Encoding: 7bit Content-Type: text/plain > On Sun, 25 Feb 2024, at 16:14, S.N.Grigoriev wrote: > >>> Another thing maybe worthwhile trying is to just have that one thing >>> plugged into the usb subsystem. Beforehand, tail -f /var/log/messages >>> in an xterm, unplug everything usb (2&3), plug in the cd drive, >>> observe messages. >> >> I've done it. The same result. > > What is the output of lsusb -v ? > > lsusb output in the file attached. The following error message displayed during execution: can't get debug descriptor: Input/output error. Does it mean that my USB hardware is broken? Regards, Serguey. ------==--bound.487476.omv3tj5zwvd3plqc.iva.yp-c.yandex.net Content-Disposition: attachment; filename="lsusb.txt" Content-Transfer-Encoding: base64 Content-Type: text/plain; name="lsusb.txt" CkJ1cyAvZGV2L3VzYiBEZXZpY2UgL2Rldi91Z2VuMi4zOiBJRCAxYTJjOjBjMjEgQ2hpbmEgUmVz b3VyY2UgU2VtaWNvIENvLiwgTHRkIApEZXZpY2UgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAg ICAgICAgICAgIDE4CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgMQogIGJjZFVTQiAgICAgICAg ICAgICAgIDEuMTAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICAwIChEZWZpbmVkIGF0IEludGVy ZmFjZSBsZXZlbCkKICBiRGV2aWNlU3ViQ2xhc3MgICAgICAgICAwIAogIGJEZXZpY2VQcm90b2Nv bCAgICAgICAgIDAgCiAgYk1heFBhY2tldFNpemUwICAgICAgICAgOAogIGlkVmVuZG9yICAgICAg ICAgICAweDFhMmMgQ2hpbmEgUmVzb3VyY2UgU2VtaWNvIENvLiwgTHRkCiAgaWRQcm9kdWN0ICAg ICAgICAgIDB4MGMyMSAKICBiY2REZXZpY2UgICAgICAgICAgICAxLjEwCiAgaU1hbnVmYWN0dXJl ciAgICAgICAgICAgMSBVU0IKICBpUHJvZHVjdCAgICAgICAgICAgICAgICAyIFVTQiBLZXlib2Fy ZAogIGlTZXJpYWwgICAgICAgICAgICAgICAgIDAgCiAgYk51bUNvbmZpZ3VyYXRpb25zICAgICAg MQogIENvbmZpZ3VyYXRpb24gRGVzY3JpcHRvcjoKICAgIGJMZW5ndGggICAgICAgICAgICAgICAg IDkKICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDIKICAgIHdUb3RhbExlbmd0aCAgICAgICAg ICAgNTkKICAgIGJOdW1JbnRlcmZhY2VzICAgICAgICAgIDIKICAgIGJDb25maWd1cmF0aW9uVmFs dWUgICAgIDEKICAgIGlDb25maWd1cmF0aW9uICAgICAgICAgIDAgCiAgICBibUF0dHJpYnV0ZXMg ICAgICAgICAweGEwCiAgICAgIChCdXMgUG93ZXJlZCkKICAgICAgUmVtb3RlIFdha2V1cAogICAg TWF4UG93ZXIgICAgICAgICAgICAgIDEwMG1BCiAgICBJbnRlcmZhY2UgRGVzY3JpcHRvcjoKICAg ICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgICBiRGVzY3JpcHRvclR5cGUgICAgICAg ICA0CiAgICAgIGJJbnRlcmZhY2VOdW1iZXIgICAgICAgIDAKICAgICAgYkFsdGVybmF0ZVNldHRp bmcgICAgICAgMAogICAgICBiTnVtRW5kcG9pbnRzICAgICAgICAgICAxCiAgICAgIGJJbnRlcmZh Y2VDbGFzcyAgICAgICAgIDMgSHVtYW4gSW50ZXJmYWNlIERldmljZQogICAgICBiSW50ZXJmYWNl U3ViQ2xhc3MgICAgICAxIEJvb3QgSW50ZXJmYWNlIFN1YmNsYXNzCiAgICAgIGJJbnRlcmZhY2VQ cm90b2NvbCAgICAgIDEgS2V5Ym9hcmQKICAgICAgaUludGVyZmFjZSAgICAgICAgICAgICAgMCAK ICAgICAgICBISUQgRGV2aWNlIERlc2NyaXB0b3I6CiAgICAgICAgICBiTGVuZ3RoICAgICAgICAg ICAgICAgICA5CiAgICAgICAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgIDMzCiAgICAgICAgICBi Y2RISUQgICAgICAgICAgICAgICAxLjEwCiAgICAgICAgICBiQ291bnRyeUNvZGUgICAgICAgICAg ICAwIE5vdCBzdXBwb3J0ZWQKICAgICAgICAgIGJOdW1EZXNjcmlwdG9ycyAgICAgICAgIDEKICAg ICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgMzQgUmVwb3J0CiAgICAgICAgICB3RGVzY3Jp cHRvckxlbmd0aCAgICAgIDU0CiAgICAgICAgICBSZXBvcnQgRGVzY3JpcHRvcjogKGxlbmd0aCBp cyA1NCkKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBVc2FnZSBQYWdlLCBkYXRhPSBbIDB4MDEg XSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBHZW5lcmljIERlc2t0b3AgQ29udHJvbHMK ICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSwgZGF0YT0gWyAweDA2IF0gNgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgS2V5Ym9hcmQKICAgICAgICAgICAgSXRlbShNYWluICApOiBD b2xsZWN0aW9uLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBB cHBsaWNhdGlvbgogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2UsIGRhdGE9IFsg MHgwOCBdIDgKICAgICAgICAgICAgICAgICAgICAgICAgICAgIExFRHMKICAgICAgICAgICAgSXRl bShMb2NhbCApOiBVc2FnZSBNaW5pbXVtLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICBOdW1Mb2NrCiAgICAgICAgICAgIEl0ZW0oTG9jYWwgKTogVXNhZ2UgTWF4 aW11bSwgZGF0YT0gWyAweDAzIF0gMwogICAgICAgICAgICAgICAgICAgICAgICAgICAgU2Nyb2xs IExvY2sKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1pbmltdW0sIGRhdGE9IFsg MHgwMCBdIDAKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1heGltdW0sIGRhdGE9 IFsgMHgwMSBdIDEKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQgU2l6ZSwgZGF0YT0g WyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0g WyAweDAzIF0gMwogICAgICAgICAgICBJdGVtKE1haW4gICk6IE91dHB1dCwgZGF0YT0gWyAweDAy IF0gMgogICAgICAgICAgICAgICAgICAgICAgICAgICAgRGF0YSBWYXJpYWJsZSBBYnNvbHV0ZSBO b19XcmFwIExpbmVhcgogICAgICAgICAgICAgICAgICAgICAgICAgICAgUHJlZmVycmVkX1N0YXRl IE5vX051bGxfUG9zaXRpb24gTm9uX1ZvbGF0aWxlIEJpdGZpZWxkCiAgICAgICAgICAgIEl0ZW0o R2xvYmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDUgXSA1CiAgICAgICAgICAgIEl0ZW0o TWFpbiAgKTogT3V0cHV0LCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICBDb25zdGFudCBBcnJheSBBYnNvbHV0ZSBOb19XcmFwIExpbmVhcgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgUHJlZmVycmVkX1N0YXRlIE5vX051bGxfUG9zaXRpb24gTm9uX1ZvbGF0 aWxlIEJpdGZpZWxkCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogVXNhZ2UgUGFnZSwgZGF0YT0g WyAweDA3IF0gNwogICAgICAgICAgICAgICAgICAgICAgICAgICAgS2V5Ym9hcmQKICAgICAgICAg ICAgSXRlbShMb2NhbCApOiBVc2FnZSBNaW5pbXVtLCBkYXRhPSBbIDB4ZTAgXSAyMjQKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIENvbnRyb2wgTGVmdAogICAgICAgICAgICBJdGVtKExvY2Fs ICk6IFVzYWdlIE1heGltdW0sIGRhdGE9IFsgMHhlNyBdIDIzMQogICAgICAgICAgICAgICAgICAg ICAgICAgICAgR1VJIFJpZ2h0CiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50 LCBkYXRhPSBbIDB4MDggXSA4CiAgICAgICAgICAgIEl0ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9 IFsgMHgwMiBdIDIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIERhdGEgVmFyaWFibGUgQWJz b2x1dGUgTm9fV3JhcCBMaW5lYXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFByZWZlcnJl ZF9TdGF0ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9Wb2xhdGlsZSBCaXRmaWVsZAogICAgICAgICAg ICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBTaXplLCBkYXRhPSBbIDB4MDggXSA4CiAgICAgICAgICAg IEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAg IEl0ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIENvbnN0YW50IEFycmF5IEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlvbiBOb25f Vm9sYXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSBNaW5pbXVt LCBkYXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBObyBFdmVudAog ICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlIE1heGltdW0sIGRhdGE9IFsgMHg5MSBdIDE0 NQogICAgICAgICAgICAgICAgICAgICAgICAgICAgTEFORyAyIChIYW5qYSBDb252ZXJzaW9uLCBL b3JlYSkKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1heGltdW0sIGRhdGE9IFsg MHhmZiAweDAwIF0gMjU1CiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBk YXRhPSBbIDB4MDYgXSA2CiAgICAgICAgICAgIEl0ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9IFsg MHgwMCBdIDAKICAgICAgICAgICAgICAgICAgICAgICAgICAgIERhdGEgQXJyYXkgQWJzb2x1dGUg Tm9fV3JhcCBMaW5lYXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFByZWZlcnJlZF9TdGF0 ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9Wb2xhdGlsZSBCaXRmaWVsZAogICAgICAgICAgICBJdGVt KE1haW4gICk6IEVuZCBDb2xsZWN0aW9uLCBkYXRhPW5vbmUKICAgICAgRW5kcG9pbnQgRGVzY3Jp cHRvcjoKICAgICAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA3CiAgICAgICAgYkRlc2NyaXB0 b3JUeXBlICAgICAgICAgNQogICAgICAgIGJFbmRwb2ludEFkZHJlc3MgICAgIDB4ODEgIEVQIDEg SU4KICAgICAgICBibUF0dHJpYnV0ZXMgICAgICAgICAgICAzCiAgICAgICAgICBUcmFuc2ZlciBU eXBlICAgICAgICAgICAgSW50ZXJydXB0CiAgICAgICAgICBTeW5jaCBUeXBlICAgICAgICAgICAg ICAgTm9uZQogICAgICAgICAgVXNhZ2UgVHlwZSAgICAgICAgICAgICAgIERhdGEKICAgICAgICB3 TWF4UGFja2V0U2l6ZSAgICAgMHgwMDA4ICAxeCA4IGJ5dGVzCiAgICAgICAgYkludGVydmFsICAg ICAgICAgICAgICAxMAogICAgSW50ZXJmYWNlIERlc2NyaXB0b3I6CiAgICAgIGJMZW5ndGggICAg ICAgICAgICAgICAgIDkKICAgICAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgNAogICAgICBiSW50 ZXJmYWNlTnVtYmVyICAgICAgICAxCiAgICAgIGJBbHRlcm5hdGVTZXR0aW5nICAgICAgIDAKICAg ICAgYk51bUVuZHBvaW50cyAgICAgICAgICAgMQogICAgICBiSW50ZXJmYWNlQ2xhc3MgICAgICAg ICAzIEh1bWFuIEludGVyZmFjZSBEZXZpY2UKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAg MSBCb290IEludGVyZmFjZSBTdWJjbGFzcwogICAgICBiSW50ZXJmYWNlUHJvdG9jb2wgICAgICAy IE1vdXNlCiAgICAgIGlJbnRlcmZhY2UgICAgICAgICAgICAgIDAgCiAgICAgICAgSElEIERldmlj ZSBEZXNjcmlwdG9yOgogICAgICAgICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgICAg ICAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAzMwogICAgICAgICAgYmNkSElEICAgICAgICAgICAg ICAgMS4xMAogICAgICAgICAgYkNvdW50cnlDb2RlICAgICAgICAgICAgMCBOb3Qgc3VwcG9ydGVk CiAgICAgICAgICBiTnVtRGVzY3JpcHRvcnMgICAgICAgICAxCiAgICAgICAgICBiRGVzY3JpcHRv clR5cGUgICAgICAgIDM0IFJlcG9ydAogICAgICAgICAgd0Rlc2NyaXB0b3JMZW5ndGggICAgIDEw OAogICAgICAgICAgUmVwb3J0IERlc2NyaXB0b3I6IChsZW5ndGggaXMgMTA4KQogICAgICAgICAg ICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2UsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAg ICAgICAgICAgICAgICAgIEdlbmVyaWMgRGVza3RvcCBDb250cm9scwogICAgICAgICAgICBJdGVt KExvY2FsICk6IFVzYWdlLCBkYXRhPSBbIDB4MDIgXSAyCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICBNb3VzZQogICAgICAgICAgICBJdGVtKE1haW4gICk6IENvbGxlY3Rpb24sIGRhdGE9IFsg MHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIEFwcGxpY2F0aW9uCiAgICAgICAg ICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IElELCBkYXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAg IEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIFBvaW50ZXIKICAgICAgICAgICAgSXRlbShNYWluICApOiBDb2xsZWN0aW9uLCBk YXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBQaHlzaWNhbAogICAg ICAgICAgICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2UsIGRhdGE9IFsgMHgwOSBdIDkKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIEJ1dHRvbnMKICAgICAgICAgICAgSXRlbShMb2NhbCApOiBV c2FnZSBNaW5pbXVtLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICBCdXR0b24gMSAoUHJpbWFyeSkKICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSBNYXhp bXVtLCBkYXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBCdXR0b24g MyAoVGVydGlhcnkpCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogTG9naWNhbCBNaW5pbXVtLCBk YXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogTG9naWNhbCBNYXhpbXVt LCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50 LCBkYXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IFNpemUs IGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgSXRlbShNYWluICApOiBJbnB1dCwgZGF0YT0g WyAweDAyIF0gMgogICAgICAgICAgICAgICAgICAgICAgICAgICAgRGF0YSBWYXJpYWJsZSBBYnNv bHV0ZSBOb19XcmFwIExpbmVhcgogICAgICAgICAgICAgICAgICAgICAgICAgICAgUHJlZmVycmVk X1N0YXRlIE5vX051bGxfUG9zaXRpb24gTm9uX1ZvbGF0aWxlIEJpdGZpZWxkCiAgICAgICAgICAg IEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAg IEl0ZW0oR2xvYmFsKTogUmVwb3J0IFNpemUsIGRhdGE9IFsgMHgwNSBdIDUKICAgICAgICAgICAg SXRlbShNYWluICApOiBJbnB1dCwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICAgICAgICAg ICAgICAgICAgQ29uc3RhbnQgQXJyYXkgQWJzb2x1dGUgTm9fV3JhcCBMaW5lYXIKICAgICAgICAg ICAgICAgICAgICAgICAgICAgIFByZWZlcnJlZF9TdGF0ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9W b2xhdGlsZSBCaXRmaWVsZAogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2UsIGRh dGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIEdlbmVyaWMgRGVza3Rv cCBDb250cm9scwogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlLCBkYXRhPSBbIDB4MzAg XSA0OAogICAgICAgICAgICAgICAgICAgICAgICAgICAgRGlyZWN0aW9uLVgKICAgICAgICAgICAg SXRlbShMb2NhbCApOiBVc2FnZSwgZGF0YT0gWyAweDMxIF0gNDkKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIERpcmVjdGlvbi1ZCiAgICAgICAgICAgIEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRh dGE9IFsgMHgzOCBdIDU2CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBXaGVlbAogICAgICAg ICAgICBJdGVtKEdsb2JhbCk6IExvZ2ljYWwgTWluaW11bSwgZGF0YT0gWyAweDgxIF0gMTI5CiAg ICAgICAgICAgIEl0ZW0oR2xvYmFsKTogTG9naWNhbCBNYXhpbXVtLCBkYXRhPSBbIDB4N2YgXSAx MjcKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQgU2l6ZSwgZGF0YT0gWyAweDA4IF0g OAogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0gWyAweDAzIF0g MwogICAgICAgICAgICBJdGVtKE1haW4gICk6IElucHV0LCBkYXRhPSBbIDB4MDYgXSA2CiAgICAg ICAgICAgICAgICAgICAgICAgICAgICBEYXRhIFZhcmlhYmxlIFJlbGF0aXZlIE5vX1dyYXAgTGlu ZWFyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Q b3NpdGlvbiBOb25fVm9sYXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShNYWluICApOiBF bmQgQ29sbGVjdGlvbiwgZGF0YT1ub25lCiAgICAgICAgICAgIEl0ZW0oTWFpbiAgKTogRW5kIENv bGxlY3Rpb24sIGRhdGE9bm9uZQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2Us IGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIEdlbmVyaWMgRGVz a3RvcCBDb250cm9scwogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlLCBkYXRhPSBbIDB4 ODAgXSAxMjgKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFN5c3RlbSBDb250cm9sCiAgICAg ICAgICAgIEl0ZW0oTWFpbiAgKTogQ29sbGVjdGlvbiwgZGF0YT0gWyAweDAxIF0gMQogICAgICAg ICAgICAgICAgICAgICAgICAgICAgQXBwbGljYXRpb24KICAgICAgICAgICAgSXRlbShHbG9iYWwp OiBSZXBvcnQgSUQsIGRhdGE9IFsgMHgwMiBdIDIKICAgICAgICAgICAgSXRlbShMb2NhbCApOiBV c2FnZSBNaW5pbXVtLCBkYXRhPSBbIDB4ODEgXSAxMjkKICAgICAgICAgICAgICAgICAgICAgICAg ICAgIFN5c3RlbSBQb3dlciBEb3duCiAgICAgICAgICAgIEl0ZW0oTG9jYWwgKTogVXNhZ2UgTWF4 aW11bSwgZGF0YT0gWyAweDgzIF0gMTMxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBTeXN0 ZW0gV2FrZSBVcAogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IExvZ2ljYWwgTWluaW11bSwgZGF0 YT0gWyAweDAwIF0gMAogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IExvZ2ljYWwgTWF4aW11bSwg ZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBTaXplLCBk YXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBk YXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAgIEl0ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9IFsg MHgwMiBdIDIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIERhdGEgVmFyaWFibGUgQWJzb2x1 dGUgTm9fV3JhcCBMaW5lYXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFByZWZlcnJlZF9T dGF0ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9Wb2xhdGlsZSBCaXRmaWVsZAogICAgICAgICAgICBJ dGVtKEdsb2JhbCk6IFJlcG9ydCBTaXplLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgIEl0 ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDUgXSA1CiAgICAgICAgICAgIEl0 ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAg ICAgICAgIENvbnN0YW50IEFycmF5IEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlvbiBOb25fVm9s YXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShNYWluICApOiBFbmQgQ29sbGVjdGlvbiwg ZGF0YT1ub25lCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogVXNhZ2UgUGFnZSwgZGF0YT0gWyAw eDBjIF0gMTIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIENvbnN1bWVyCiAgICAgICAgICAg IEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIENvbnN1bWVyIENvbnRyb2wKICAgICAgICAgICAgSXRlbShNYWluICApOiBDb2xs ZWN0aW9uLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBBcHBs aWNhdGlvbgogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBJRCwgZGF0YT0gWyAweDAx IF0gMQogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlIE1pbmltdW0sIGRhdGE9IFsgMHgw MCBdIDAKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFVuYXNzaWduZWQKICAgICAgICAgICAg SXRlbShMb2NhbCApOiBVc2FnZSBNYXhpbXVtLCBkYXRhPSBbIDB4M2MgMHgwMiBdIDU3MgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgQUMgRm9ybWF0CiAgICAgICAgICAgIEl0ZW0oR2xvYmFs KTogTG9naWNhbCBNaW5pbXVtLCBkYXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgIEl0ZW0oR2xv YmFsKTogTG9naWNhbCBNYXhpbXVtLCBkYXRhPSBbIDB4M2MgMHgwMiBdIDU3MgogICAgICAgICAg ICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAg ICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBTaXplLCBkYXRhPSBbIDB4MTAgXSAxNgogICAgICAgICAg ICBJdGVtKE1haW4gICk6IElucHV0LCBkYXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICBEYXRhIEFycmF5IEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlvbiBOb25fVm9s YXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShNYWluICApOiBFbmQgQ29sbGVjdGlvbiwg ZGF0YT1ub25lCiAgICAgIEVuZHBvaW50IERlc2NyaXB0b3I6CiAgICAgICAgYkxlbmd0aCAgICAg ICAgICAgICAgICAgNwogICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDUKICAgICAgICBi RW5kcG9pbnRBZGRyZXNzICAgICAweDgyICBFUCAyIElOCiAgICAgICAgYm1BdHRyaWJ1dGVzICAg ICAgICAgICAgMwogICAgICAgICAgVHJhbnNmZXIgVHlwZSAgICAgICAgICAgIEludGVycnVwdAog ICAgICAgICAgU3luY2ggVHlwZSAgICAgICAgICAgICAgIE5vbmUKICAgICAgICAgIFVzYWdlIFR5 cGUgICAgICAgICAgICAgICBEYXRhCiAgICAgICAgd01heFBhY2tldFNpemUgICAgIDB4MDAwNSAg MXggNSBieXRlcwogICAgICAgIGJJbnRlcnZhbCAgICAgICAgICAgICAgMTAKRGV2aWNlIFN0YXR1 czogICAgIDB4MDAwMAogIChCdXMgUG93ZXJlZCkKCkJ1cyAvZGV2L3VzYiBEZXZpY2UgL2Rldi91 Z2VuMi4yOiBJRCAwNDVlOjAwNDAgTWljcm9zb2Z0IENvcnAuIFdoZWVsIE1vdXNlIE9wdGljYWwK RGV2aWNlIERlc2NyaXB0b3I6CiAgYkxlbmd0aCAgICAgICAgICAgICAgICAxOAogIGJEZXNjcmlw dG9yVHlwZSAgICAgICAgIDEKICBiY2RVU0IgICAgICAgICAgICAgICAxLjEwCiAgYkRldmljZUNs YXNzICAgICAgICAgICAgMCAoRGVmaW5lZCBhdCBJbnRlcmZhY2UgbGV2ZWwpCiAgYkRldmljZVN1 YkNsYXNzICAgICAgICAgMCAKICBiRGV2aWNlUHJvdG9jb2wgICAgICAgICAwIAogIGJNYXhQYWNr ZXRTaXplMCAgICAgICAgIDgKICBpZFZlbmRvciAgICAgICAgICAgMHgwNDVlIE1pY3Jvc29mdCBD b3JwLgogIGlkUHJvZHVjdCAgICAgICAgICAweDAwNDAgV2hlZWwgTW91c2UgT3B0aWNhbAogIGJj ZERldmljZSAgICAgICAgICAgIDMuMDAKICBpTWFudWZhY3R1cmVyICAgICAgICAgICAxIE1pY3Jv c29mdAogIGlQcm9kdWN0ICAgICAgICAgICAgICAgIDMgTWljcm9zb2Z0IDMtQnV0dG9uIE1vdXNl IHdpdGggSW50ZWxsaUV5ZShUTSkKICBpU2VyaWFsICAgICAgICAgICAgICAgICAwIAogIGJOdW1D b25maWd1cmF0aW9ucyAgICAgIDEKICBDb25maWd1cmF0aW9uIERlc2NyaXB0b3I6CiAgICBiTGVu Z3RoICAgICAgICAgICAgICAgICA5CiAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgICAyCiAgICB3 VG90YWxMZW5ndGggICAgICAgICAgIDM0CiAgICBiTnVtSW50ZXJmYWNlcyAgICAgICAgICAxCiAg ICBiQ29uZmlndXJhdGlvblZhbHVlICAgICAxCiAgICBpQ29uZmlndXJhdGlvbiAgICAgICAgICAw IAogICAgYm1BdHRyaWJ1dGVzICAgICAgICAgMHhhMAogICAgICAoQnVzIFBvd2VyZWQpCiAgICAg IFJlbW90ZSBXYWtldXAKICAgIE1heFBvd2VyICAgICAgICAgICAgICAxMDBtQQogICAgSW50ZXJm YWNlIERlc2NyaXB0b3I6CiAgICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDkKICAgICAgYkRl c2NyaXB0b3JUeXBlICAgICAgICAgNAogICAgICBiSW50ZXJmYWNlTnVtYmVyICAgICAgICAwCiAg ICAgIGJBbHRlcm5hdGVTZXR0aW5nICAgICAgIDAKICAgICAgYk51bUVuZHBvaW50cyAgICAgICAg ICAgMQogICAgICBiSW50ZXJmYWNlQ2xhc3MgICAgICAgICAzIEh1bWFuIEludGVyZmFjZSBEZXZp Y2UKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAgMSBCb290IEludGVyZmFjZSBTdWJjbGFz cwogICAgICBiSW50ZXJmYWNlUHJvdG9jb2wgICAgICAyIE1vdXNlCiAgICAgIGlJbnRlcmZhY2Ug ICAgICAgICAgICAgIDAgCiAgICAgICAgSElEIERldmljZSBEZXNjcmlwdG9yOgogICAgICAgICAg Ykxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgICAgICAgYkRlc2NyaXB0b3JUeXBlICAgICAg ICAzMwogICAgICAgICAgYmNkSElEICAgICAgICAgICAgICAgMS4xMAogICAgICAgICAgYkNvdW50 cnlDb2RlICAgICAgICAgICAgMCBOb3Qgc3VwcG9ydGVkCiAgICAgICAgICBiTnVtRGVzY3JpcHRv cnMgICAgICAgICAxCiAgICAgICAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgIDM0IFJlcG9ydAog ICAgICAgICAgd0Rlc2NyaXB0b3JMZW5ndGggICAgICA3MgogICAgICAgICAgUmVwb3J0IERlc2Ny aXB0b3I6IChsZW5ndGggaXMgNzIpCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogVXNhZ2UgUGFn ZSwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgR2VuZXJpYyBE ZXNrdG9wIENvbnRyb2xzCiAgICAgICAgICAgIEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRhdGE9IFsg MHgwMiBdIDIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIE1vdXNlCiAgICAgICAgICAgIEl0 ZW0oTWFpbiAgKTogQ29sbGVjdGlvbiwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICAgICAg ICAgICAgICAgICAgQXBwbGljYXRpb24KICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSwg ZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgUG9pbnRlcgogICAg ICAgICAgICBJdGVtKE1haW4gICk6IENvbGxlY3Rpb24sIGRhdGE9IFsgMHgwMCBdIDAKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIFBoeXNpY2FsCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTog VXNhZ2UgUGFnZSwgZGF0YT0gWyAweDA5IF0gOQogICAgICAgICAgICAgICAgICAgICAgICAgICAg QnV0dG9ucwogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlIE1pbmltdW0sIGRhdGE9IFsg MHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIEJ1dHRvbiAxIChQcmltYXJ5KQog ICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlIE1heGltdW0sIGRhdGE9IFsgMHgwMyBdIDMK ICAgICAgICAgICAgICAgICAgICAgICAgICAgIEJ1dHRvbiAzIChUZXJ0aWFyeSkKICAgICAgICAg ICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1pbmltdW0sIGRhdGE9IFsgMHgwMCBdIDAKICAgICAg ICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1heGltdW0sIGRhdGE9IFsgMHgwMSBdIDEKICAg ICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQgU2l6ZSwgZGF0YT0gWyAweDAxIF0gMQogICAg ICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0gWyAweDAzIF0gMwogICAg ICAgICAgICBJdGVtKE1haW4gICk6IElucHV0LCBkYXRhPSBbIDB4MDIgXSAyCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICBEYXRhIFZhcmlhYmxlIEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlv biBOb25fVm9sYXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQg U2l6ZSwgZGF0YT0gWyAweDA1IF0gNQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBD b3VudCwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKE1haW4gICk6IElucHV0LCBk YXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBDb25zdGFudCBBcnJh eSBBYnNvbHV0ZSBOb19XcmFwIExpbmVhcgogICAgICAgICAgICAgICAgICAgICAgICAgICAgUHJl ZmVycmVkX1N0YXRlIE5vX051bGxfUG9zaXRpb24gTm9uX1ZvbGF0aWxlIEJpdGZpZWxkCiAgICAg ICAgICAgIEl0ZW0oR2xvYmFsKTogVXNhZ2UgUGFnZSwgZGF0YT0gWyAweDAxIF0gMQogICAgICAg ICAgICAgICAgICAgICAgICAgICAgR2VuZXJpYyBEZXNrdG9wIENvbnRyb2xzCiAgICAgICAgICAg IEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRhdGE9IFsgMHgzMCBdIDQ4CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICBEaXJlY3Rpb24tWAogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlLCBk YXRhPSBbIDB4MzEgXSA0OQogICAgICAgICAgICAgICAgICAgICAgICAgICAgRGlyZWN0aW9uLVkK ICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSwgZGF0YT0gWyAweDM4IF0gNTYKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIFdoZWVsCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogTG9n aWNhbCBNaW5pbXVtLCBkYXRhPSBbIDB4ODEgXSAxMjkKICAgICAgICAgICAgSXRlbShHbG9iYWwp OiBMb2dpY2FsIE1heGltdW0sIGRhdGE9IFsgMHg3ZiBdIDEyNwogICAgICAgICAgICBJdGVtKEds b2JhbCk6IFJlcG9ydCBTaXplLCBkYXRhPSBbIDB4MDggXSA4CiAgICAgICAgICAgIEl0ZW0oR2xv YmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAgIEl0ZW0oTWFp biAgKTogSW5wdXQsIGRhdGE9IFsgMHgwNiBdIDYKICAgICAgICAgICAgICAgICAgICAgICAgICAg IERhdGEgVmFyaWFibGUgUmVsYXRpdmUgTm9fV3JhcCBMaW5lYXIKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIFByZWZlcnJlZF9TdGF0ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9Wb2xhdGlsZSBC aXRmaWVsZAogICAgICAgICAgICBJdGVtKE1haW4gICk6IEVuZCBDb2xsZWN0aW9uLCBkYXRhPW5v bmUKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBVc2FnZSBQYWdlLCBkYXRhPSBbIDB4ZmYgXSAy NTUKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFZlbmRvciBTcGVjaWZpYwogICAgICAgICAg ICBJdGVtKExvY2FsICk6IFVzYWdlLCBkYXRhPSBbIDB4MDIgXSAyCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAobnVsbCkKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1pbmlt dW0sIGRhdGE9IFsgMHgwMCBdIDAKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1h eGltdW0sIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQg U2l6ZSwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBD b3VudCwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKE1haW4gICk6IEZlYXR1cmUs IGRhdGE9IFsgMHgyMiBdIDM0CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBEYXRhIFZhcmlh YmxlIEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBO b19QcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlvbiBOb25fVm9sYXRpbGUgQml0ZmllbGQK ICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQgU2l6ZSwgZGF0YT0gWyAweDA3IF0gNwog ICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0gWyAweDAxIF0gMQog ICAgICAgICAgICBJdGVtKE1haW4gICk6IEZlYXR1cmUsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIENvbnN0YW50IEFycmF5IEFic29sdXRlIE5vX1dyYXAgTGlu ZWFyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Q b3NpdGlvbiBOb25fVm9sYXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShNYWluICApOiBF bmQgQ29sbGVjdGlvbiwgZGF0YT1ub25lCiAgICAgIEVuZHBvaW50IERlc2NyaXB0b3I6CiAgICAg ICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgNwogICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAg ICAgIDUKICAgICAgICBiRW5kcG9pbnRBZGRyZXNzICAgICAweDgxICBFUCAxIElOCiAgICAgICAg Ym1BdHRyaWJ1dGVzICAgICAgICAgICAgMwogICAgICAgICAgVHJhbnNmZXIgVHlwZSAgICAgICAg ICAgIEludGVycnVwdAogICAgICAgICAgU3luY2ggVHlwZSAgICAgICAgICAgICAgIE5vbmUKICAg ICAgICAgIFVzYWdlIFR5cGUgICAgICAgICAgICAgICBEYXRhCiAgICAgICAgd01heFBhY2tldFNp emUgICAgIDB4MDAwNCAgMXggNCBieXRlcwogICAgICAgIGJJbnRlcnZhbCAgICAgICAgICAgICAg MTAKRGV2aWNlIFN0YXR1czogICAgIDB4MDAwMAogIChCdXMgUG93ZXJlZCkKCkJ1cyAvZGV2L3Vz YiBEZXZpY2UgL2Rldi91Z2VuMC4xOiBJRCAwMDAwOjAwMDAgIApEZXZpY2UgRGVzY3JpcHRvcjoK ICBiTGVuZ3RoICAgICAgICAgICAgICAgIDE4CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgMQog IGJjZFVTQiAgICAgICAgICAgICAgIDEuMDAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1 YgogIGJEZXZpY2VTdWJDbGFzcyAgICAgICAgIDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAg ICAgICAgMCBGdWxsIHNwZWVkIChvciByb290KSBodWIKICBiTWF4UGFja2V0U2l6ZTAgICAgICAg IDY0CiAgaWRWZW5kb3IgICAgICAgICAgIDB4MDAwMCAKICBpZFByb2R1Y3QgICAgICAgICAgMHgw MDAwIAogIGJjZERldmljZSAgICAgICAgICAgIDEuMDAKICBpTWFudWZhY3R1cmVyICAgICAgICAg ICAxIEFUSQogIGlQcm9kdWN0ICAgICAgICAgICAgICAgIDIgT0hDSSByb290IEhVQgogIGlTZXJp YWwgICAgICAgICAgICAgICAgIDAgCiAgYk51bUNvbmZpZ3VyYXRpb25zICAgICAgMQogIENvbmZp Z3VyYXRpb24gRGVzY3JpcHRvcjoKICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDkKICAgIGJE ZXNjcmlwdG9yVHlwZSAgICAgICAgIDIKICAgIHdUb3RhbExlbmd0aCAgICAgICAgICAgMjUKICAg IGJOdW1JbnRlcmZhY2VzICAgICAgICAgIDEKICAgIGJDb25maWd1cmF0aW9uVmFsdWUgICAgIDEK ICAgIGlDb25maWd1cmF0aW9uICAgICAgICAgIDAgCiAgICBibUF0dHJpYnV0ZXMgICAgICAgICAw eDQwCiAgICAgIChNaXNzaW5nIG11c3QtYmUtc2V0IGJpdCEpCiAgICAgIFNlbGYgUG93ZXJlZAog ICAgTWF4UG93ZXIgICAgICAgICAgICAgICAgMG1BCiAgICBJbnRlcmZhY2UgRGVzY3JpcHRvcjoK ICAgICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgICBiRGVzY3JpcHRvclR5cGUgICAg ICAgICA0CiAgICAgIGJJbnRlcmZhY2VOdW1iZXIgICAgICAgIDAKICAgICAgYkFsdGVybmF0ZVNl dHRpbmcgICAgICAgMAogICAgICBiTnVtRW5kcG9pbnRzICAgICAgICAgICAxCiAgICAgIGJJbnRl cmZhY2VDbGFzcyAgICAgICAgIDkgSHViCiAgICAgIGJJbnRlcmZhY2VTdWJDbGFzcyAgICAgIDAg VW51c2VkCiAgICAgIGJJbnRlcmZhY2VQcm90b2NvbCAgICAgIDAgRnVsbCBzcGVlZCAob3Igcm9v dCkgaHViCiAgICAgIGlJbnRlcmZhY2UgICAgICAgICAgICAgIDAgCiAgICAgIEVuZHBvaW50IERl c2NyaXB0b3I6CiAgICAgICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgNwogICAgICAgIGJEZXNj cmlwdG9yVHlwZSAgICAgICAgIDUKICAgICAgICBiRW5kcG9pbnRBZGRyZXNzICAgICAweDgxICBF UCAxIElOCiAgICAgICAgYm1BdHRyaWJ1dGVzICAgICAgICAgICAgMwogICAgICAgICAgVHJhbnNm ZXIgVHlwZSAgICAgICAgICAgIEludGVycnVwdAogICAgICAgICAgU3luY2ggVHlwZSAgICAgICAg ICAgICAgIE5vbmUKICAgICAgICAgIFVzYWdlIFR5cGUgICAgICAgICAgICAgICBEYXRhCiAgICAg ICAgd01heFBhY2tldFNpemUgICAgIDB4MDAyMCAgMXggMzIgYnl0ZXMKICAgICAgICBiSW50ZXJ2 YWwgICAgICAgICAgICAgMjU1Ckh1YiBEZXNjcmlwdG9yOgogIGJMZW5ndGggICAgICAgICAgICAg ICA5CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgNDEKICBuTmJyUG9ydHMgICAgICAgICAgICAgNQog IHdIdWJDaGFyYWN0ZXJpc3RpYyAweDAwMDAKICAgIEdhbmdlZCBwb3dlciBzd2l0Y2hpbmcKICAg IEdhbmdlZCBvdmVyY3VycmVudCBwcm90ZWN0aW9uCiAgYlB3ck9uMlB3ckdvb2QgICAgICAgIDIg KiAyIG1pbGxpIHNlY29uZHMKICBiSHViQ29udHJDdXJyZW50ICAgICAgMCBtaWxsaSBBbXBlcmUK ICBEZXZpY2VSZW1vdmFibGUgICAgMHgwMAogIFBvcnRQd3JDdHJsTWFzayAgICAweDAwCiBIdWIg UG9ydCBTdGF0dXM6CiAgIFBvcnQgMTogMDAwMC4wMTAwIHBvd2VyCiAgIFBvcnQgMjogMDAwMC4w MTAwIHBvd2VyCiAgIFBvcnQgMzogMDAwMC4wMTAwIHBvd2VyCiAgIFBvcnQgNDogMDAwMC4wMTAw IHBvd2VyCiAgIFBvcnQgNTogMDAwMC4wMTAwIHBvd2VyCkRldmljZSBTdGF0dXM6ICAgICAweDAw MDEKICBTZWxmIFBvd2VyZWQKCkJ1cyAvZGV2L3VzYiBEZXZpY2UgL2Rldi91Z2VuNC4xOiBJRCAw MDAwOjAwMDAgIApEZXZpY2UgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAgICAgICAgICAgIDE4 CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgMQogIGJjZFVTQiAgICAgICAgICAgICAgIDEuMDAK ICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAgICAgICAg IDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAgICAgICAgMCBGdWxsIHNwZWVkIChvciByb290 KSBodWIKICBiTWF4UGFja2V0U2l6ZTAgICAgICAgIDY0CiAgaWRWZW5kb3IgICAgICAgICAgIDB4 MDAwMCAKICBpZFByb2R1Y3QgICAgICAgICAgMHgwMDAwIAogIGJjZERldmljZSAgICAgICAgICAg IDEuMDAKICBpTWFudWZhY3R1cmVyICAgICAgICAgICAxIEFUSQogIGlQcm9kdWN0ICAgICAgICAg ICAgICAgIDIgT0hDSSByb290IEhVQgogIGlTZXJpYWwgICAgICAgICAgICAgICAgIDAgCiAgYk51 bUNvbmZpZ3VyYXRpb25zICAgICAgMQogIENvbmZpZ3VyYXRpb24gRGVzY3JpcHRvcjoKICAgIGJM ZW5ndGggICAgICAgICAgICAgICAgIDkKICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDIKICAg IHdUb3RhbExlbmd0aCAgICAgICAgICAgMjUKICAgIGJOdW1JbnRlcmZhY2VzICAgICAgICAgIDEK ICAgIGJDb25maWd1cmF0aW9uVmFsdWUgICAgIDEKICAgIGlDb25maWd1cmF0aW9uICAgICAgICAg IDAgCiAgICBibUF0dHJpYnV0ZXMgICAgICAgICAweDQwCiAgICAgIChNaXNzaW5nIG11c3QtYmUt c2V0IGJpdCEpCiAgICAgIFNlbGYgUG93ZXJlZAogICAgTWF4UG93ZXIgICAgICAgICAgICAgICAg MG1BCiAgICBJbnRlcmZhY2UgRGVzY3JpcHRvcjoKICAgICAgYkxlbmd0aCAgICAgICAgICAgICAg ICAgOQogICAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgICA0CiAgICAgIGJJbnRlcmZhY2VOdW1i ZXIgICAgICAgIDAKICAgICAgYkFsdGVybmF0ZVNldHRpbmcgICAgICAgMAogICAgICBiTnVtRW5k cG9pbnRzICAgICAgICAgICAxCiAgICAgIGJJbnRlcmZhY2VDbGFzcyAgICAgICAgIDkgSHViCiAg ICAgIGJJbnRlcmZhY2VTdWJDbGFzcyAgICAgIDAgVW51c2VkCiAgICAgIGJJbnRlcmZhY2VQcm90 b2NvbCAgICAgIDAgRnVsbCBzcGVlZCAob3Igcm9vdCkgaHViCiAgICAgIGlJbnRlcmZhY2UgICAg ICAgICAgICAgIDAgCiAgICAgIEVuZHBvaW50IERlc2NyaXB0b3I6CiAgICAgICAgYkxlbmd0aCAg ICAgICAgICAgICAgICAgNwogICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDUKICAgICAg ICBiRW5kcG9pbnRBZGRyZXNzICAgICAweDgxICBFUCAxIElOCiAgICAgICAgYm1BdHRyaWJ1dGVz ICAgICAgICAgICAgMwogICAgICAgICAgVHJhbnNmZXIgVHlwZSAgICAgICAgICAgIEludGVycnVw dAogICAgICAgICAgU3luY2ggVHlwZSAgICAgICAgICAgICAgIE5vbmUKICAgICAgICAgIFVzYWdl IFR5cGUgICAgICAgICAgICAgICBEYXRhCiAgICAgICAgd01heFBhY2tldFNpemUgICAgIDB4MDAy MCAgMXggMzIgYnl0ZXMKICAgICAgICBiSW50ZXJ2YWwgICAgICAgICAgICAgMjU1Ckh1YiBEZXNj cmlwdG9yOgogIGJMZW5ndGggICAgICAgICAgICAgICA5CiAgYkRlc2NyaXB0b3JUeXBlICAgICAg NDEKICBuTmJyUG9ydHMgICAgICAgICAgICAgMgogIHdIdWJDaGFyYWN0ZXJpc3RpYyAweDAwMDAK ICAgIEdhbmdlZCBwb3dlciBzd2l0Y2hpbmcKICAgIEdhbmdlZCBvdmVyY3VycmVudCBwcm90ZWN0 aW9uCiAgYlB3ck9uMlB3ckdvb2QgICAgICAgIDIgKiAyIG1pbGxpIHNlY29uZHMKICBiSHViQ29u dHJDdXJyZW50ICAgICAgMCBtaWxsaSBBbXBlcmUKICBEZXZpY2VSZW1vdmFibGUgICAgMHgwMAog IFBvcnRQd3JDdHJsTWFzayAgICAweDAwCiBIdWIgUG9ydCBTdGF0dXM6CiAgIFBvcnQgMTogMDAw MC4wMTAwIHBvd2VyCiAgIFBvcnQgMjogMDAwMC4wMTAwIHBvd2VyCkRldmljZSBTdGF0dXM6ICAg ICAweDAwMDEKICBTZWxmIFBvd2VyZWQKCkJ1cyAvZGV2L3VzYiBEZXZpY2UgL2Rldi91Z2VuNy4x OiBJRCAwMDAwOjAwMDAgIApEZXZpY2UgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAgICAgICAg ICAgIDE4CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgMQogIGJjZFVTQiAgICAgICAgICAgICAg IDIuMDAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAg ICAgICAgIDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAgICAgICAgMSBTaW5nbGUgVFQKICBi TWF4UGFja2V0U2l6ZTAgICAgICAgIDY0CiAgaWRWZW5kb3IgICAgICAgICAgIDB4MDAwMCAKICBp ZFByb2R1Y3QgICAgICAgICAgMHgwMDAwIAogIGJjZERldmljZSAgICAgICAgICAgIDEuMDAKICBp TWFudWZhY3R1cmVyICAgICAgICAgICAxIEFUSQogIGlQcm9kdWN0ICAgICAgICAgICAgICAgIDIg RUhDSSByb290IEhVQgogIGlTZXJpYWwgICAgICAgICAgICAgICAgIDAgCiAgYk51bUNvbmZpZ3Vy YXRpb25zICAgICAgMQogIENvbmZpZ3VyYXRpb24gRGVzY3JpcHRvcjoKICAgIGJMZW5ndGggICAg ICAgICAgICAgICAgIDkKICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDIKICAgIHdUb3RhbExl bmd0aCAgICAgICAgICAgMjUKICAgIGJOdW1JbnRlcmZhY2VzICAgICAgICAgIDEKICAgIGJDb25m aWd1cmF0aW9uVmFsdWUgICAgIDEKICAgIGlDb25maWd1cmF0aW9uICAgICAgICAgIDAgCiAgICBi bUF0dHJpYnV0ZXMgICAgICAgICAweDQwCiAgICAgIChNaXNzaW5nIG11c3QtYmUtc2V0IGJpdCEp CiAgICAgIFNlbGYgUG93ZXJlZAogICAgTWF4UG93ZXIgICAgICAgICAgICAgICAgMG1BCiAgICBJ bnRlcmZhY2UgRGVzY3JpcHRvcjoKICAgICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAg ICBiRGVzY3JpcHRvclR5cGUgICAgICAgICA0CiAgICAgIGJJbnRlcmZhY2VOdW1iZXIgICAgICAg IDAKICAgICAgYkFsdGVybmF0ZVNldHRpbmcgICAgICAgMAogICAgICBiTnVtRW5kcG9pbnRzICAg ICAgICAgICAxCiAgICAgIGJJbnRlcmZhY2VDbGFzcyAgICAgICAgIDkgSHViCiAgICAgIGJJbnRl cmZhY2VTdWJDbGFzcyAgICAgIDAgVW51c2VkCiAgICAgIGJJbnRlcmZhY2VQcm90b2NvbCAgICAg IDAgRnVsbCBzcGVlZCAob3Igcm9vdCkgaHViCiAgICAgIGlJbnRlcmZhY2UgICAgICAgICAgICAg IDAgCiAgICAgIEVuZHBvaW50IERlc2NyaXB0b3I6CiAgICAgICAgYkxlbmd0aCAgICAgICAgICAg ICAgICAgNwogICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDUKICAgICAgICBiRW5kcG9p bnRBZGRyZXNzICAgICAweDgxICBFUCAxIElOCiAgICAgICAgYm1BdHRyaWJ1dGVzICAgICAgICAg ICAgMwogICAgICAgICAgVHJhbnNmZXIgVHlwZSAgICAgICAgICAgIEludGVycnVwdAogICAgICAg ICAgU3luY2ggVHlwZSAgICAgICAgICAgICAgIE5vbmUKICAgICAgICAgIFVzYWdlIFR5cGUgICAg ICAgICAgICAgICBEYXRhCiAgICAgICAgd01heFBhY2tldFNpemUgICAgIDB4MDAwOCAgMXggOCBi eXRlcwogICAgICAgIGJJbnRlcnZhbCAgICAgICAgICAgICAyNTUKSHViIERlc2NyaXB0b3I6CiAg Ykxlbmd0aCAgICAgICAgICAgICAgIDkKICBiRGVzY3JpcHRvclR5cGUgICAgICA0MQogIG5OYnJQ b3J0cyAgICAgICAgICAgICA0CiAgd0h1YkNoYXJhY3RlcmlzdGljIDB4MDAwMgogICAgTm8gcG93 ZXIgc3dpdGNoaW5nICh1c2IgMS4wKQogICAgR2FuZ2VkIG92ZXJjdXJyZW50IHByb3RlY3Rpb24K ICAgIFRUIHRoaW5rIHRpbWUgOCBGUyBiaXRzCiAgYlB3ck9uMlB3ckdvb2QgICAgICAyMDAgKiAy IG1pbGxpIHNlY29uZHMKICBiSHViQ29udHJDdXJyZW50ICAgICAgMCBtaWxsaSBBbXBlcmUKICBE ZXZpY2VSZW1vdmFibGUgICAgMHgwMAogIFBvcnRQd3JDdHJsTWFzayAgICAweDAwCiBIdWIgUG9y dCBTdGF0dXM6CiAgIFBvcnQgMTogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgogICBQb3J0IDI6 IDAwMDAuMDUwMCBoaWdoc3BlZWQgcG93ZXIKICAgUG9ydCAzOiAwMDAwLjA1MDAgaGlnaHNwZWVk IHBvd2VyCiAgIFBvcnQgNDogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgpEZXZpY2UgUXVhbGlm aWVyIChmb3Igb3RoZXIgZGV2aWNlIHNwZWVkKToKICBiTGVuZ3RoICAgICAgICAgICAgICAgIDEw CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgNgogIGJjZFVTQiAgICAgICAgICAgICAgIDIuMDAK ICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAgICAgICAg IDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAgICAgICAgMCBGdWxsIHNwZWVkIChvciByb290 KSBodWIKICBiTWF4UGFja2V0U2l6ZTAgICAgICAgICAwCiAgYk51bUNvbmZpZ3VyYXRpb25zICAg ICAgMApEZXZpY2UgU3RhdHVzOiAgICAgMHgwMDAxCiAgU2VsZiBQb3dlcmVkCgpCdXMgL2Rldi91 c2IgRGV2aWNlIC9kZXYvdWdlbjMuMTogSUQgMDAwMDowMDAwICAKRGV2aWNlIERlc2NyaXB0b3I6 CiAgYkxlbmd0aCAgICAgICAgICAgICAgICAxOAogIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDEK ICBiY2RVU0IgICAgICAgICAgICAgICAyLjAwCiAgYkRldmljZUNsYXNzICAgICAgICAgICAgOSBI dWIKICBiRGV2aWNlU3ViQ2xhc3MgICAgICAgICAwIFVudXNlZAogIGJEZXZpY2VQcm90b2NvbCAg ICAgICAgIDEgU2luZ2xlIFRUCiAgYk1heFBhY2tldFNpemUwICAgICAgICA2NAogIGlkVmVuZG9y ICAgICAgICAgICAweDAwMDAgCiAgaWRQcm9kdWN0ICAgICAgICAgIDB4MDAwMCAKICBiY2REZXZp Y2UgICAgICAgICAgICAxLjAwCiAgaU1hbnVmYWN0dXJlciAgICAgICAgICAgMSBBVEkKICBpUHJv ZHVjdCAgICAgICAgICAgICAgICAyIEVIQ0kgcm9vdCBIVUIKICBpU2VyaWFsICAgICAgICAgICAg ICAgICAwIAogIGJOdW1Db25maWd1cmF0aW9ucyAgICAgIDEKICBDb25maWd1cmF0aW9uIERlc2Ny aXB0b3I6CiAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA5CiAgICBiRGVzY3JpcHRvclR5cGUg ICAgICAgICAyCiAgICB3VG90YWxMZW5ndGggICAgICAgICAgIDI1CiAgICBiTnVtSW50ZXJmYWNl cyAgICAgICAgICAxCiAgICBiQ29uZmlndXJhdGlvblZhbHVlICAgICAxCiAgICBpQ29uZmlndXJh dGlvbiAgICAgICAgICAwIAogICAgYm1BdHRyaWJ1dGVzICAgICAgICAgMHg0MAogICAgICAoTWlz c2luZyBtdXN0LWJlLXNldCBiaXQhKQogICAgICBTZWxmIFBvd2VyZWQKICAgIE1heFBvd2VyICAg ICAgICAgICAgICAgIDBtQQogICAgSW50ZXJmYWNlIERlc2NyaXB0b3I6CiAgICAgIGJMZW5ndGgg ICAgICAgICAgICAgICAgIDkKICAgICAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgNAogICAgICBi SW50ZXJmYWNlTnVtYmVyICAgICAgICAwCiAgICAgIGJBbHRlcm5hdGVTZXR0aW5nICAgICAgIDAK ICAgICAgYk51bUVuZHBvaW50cyAgICAgICAgICAgMQogICAgICBiSW50ZXJmYWNlQ2xhc3MgICAg ICAgICA5IEh1YgogICAgICBiSW50ZXJmYWNlU3ViQ2xhc3MgICAgICAwIFVudXNlZAogICAgICBi SW50ZXJmYWNlUHJvdG9jb2wgICAgICAwIEZ1bGwgc3BlZWQgKG9yIHJvb3QpIGh1YgogICAgICBp SW50ZXJmYWNlICAgICAgICAgICAgICAwIAogICAgICBFbmRwb2ludCBEZXNjcmlwdG9yOgogICAg ICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDcKICAgICAgICBiRGVzY3JpcHRvclR5cGUgICAg ICAgICA1CiAgICAgICAgYkVuZHBvaW50QWRkcmVzcyAgICAgMHg4MSAgRVAgMSBJTgogICAgICAg IGJtQXR0cmlidXRlcyAgICAgICAgICAgIDMKICAgICAgICAgIFRyYW5zZmVyIFR5cGUgICAgICAg ICAgICBJbnRlcnJ1cHQKICAgICAgICAgIFN5bmNoIFR5cGUgICAgICAgICAgICAgICBOb25lCiAg ICAgICAgICBVc2FnZSBUeXBlICAgICAgICAgICAgICAgRGF0YQogICAgICAgIHdNYXhQYWNrZXRT aXplICAgICAweDAwMDggIDF4IDggYnl0ZXMKICAgICAgICBiSW50ZXJ2YWwgICAgICAgICAgICAg MjU1Ckh1YiBEZXNjcmlwdG9yOgogIGJMZW5ndGggICAgICAgICAgICAgICA5CiAgYkRlc2NyaXB0 b3JUeXBlICAgICAgNDEKICBuTmJyUG9ydHMgICAgICAgICAgICAgNQogIHdIdWJDaGFyYWN0ZXJp c3RpYyAweDAwMDIKICAgIE5vIHBvd2VyIHN3aXRjaGluZyAodXNiIDEuMCkKICAgIEdhbmdlZCBv dmVyY3VycmVudCBwcm90ZWN0aW9uCiAgICBUVCB0aGluayB0aW1lIDggRlMgYml0cwogIGJQd3JP bjJQd3JHb29kICAgICAgMjAwICogMiBtaWxsaSBzZWNvbmRzCiAgYkh1YkNvbnRyQ3VycmVudCAg ICAgIDAgbWlsbGkgQW1wZXJlCiAgRGV2aWNlUmVtb3ZhYmxlICAgIDB4MDAKICBQb3J0UHdyQ3Ry bE1hc2sgICAgMHgwMAogSHViIFBvcnQgU3RhdHVzOgogICBQb3J0IDE6IDAwMDAuMDUwMCBoaWdo c3BlZWQgcG93ZXIKICAgUG9ydCAyOiAwMDAwLjA1MDAgaGlnaHNwZWVkIHBvd2VyCiAgIFBvcnQg MzogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgogICBQb3J0IDQ6IDAwMDAuMDUwMCBoaWdoc3Bl ZWQgcG93ZXIKICAgUG9ydCA1OiAwMDAwLjA1MDAgaGlnaHNwZWVkIHBvd2VyCkRldmljZSBRdWFs aWZpZXIgKGZvciBvdGhlciBkZXZpY2Ugc3BlZWQpOgogIGJMZW5ndGggICAgICAgICAgICAgICAg MTAKICBiRGVzY3JpcHRvclR5cGUgICAgICAgICA2CiAgYmNkVVNCICAgICAgICAgICAgICAgMi4w MAogIGJEZXZpY2VDbGFzcyAgICAgICAgICAgIDkgSHViCiAgYkRldmljZVN1YkNsYXNzICAgICAg ICAgMCBVbnVzZWQKICBiRGV2aWNlUHJvdG9jb2wgICAgICAgICAwIEZ1bGwgc3BlZWQgKG9yIHJv b3QpIGh1YgogIGJNYXhQYWNrZXRTaXplMCAgICAgICAgIDAKICBiTnVtQ29uZmlndXJhdGlvbnMg ICAgICAwCkRldmljZSBTdGF0dXM6ICAgICAweDAwMDEKICBTZWxmIFBvd2VyZWQKCkJ1cyAvZGV2 L3VzYiBEZXZpY2UgL2Rldi91Z2VuNS4xOiBJRCAwMDAwOjAwMDAgIApEZXZpY2UgRGVzY3JpcHRv cjoKICBiTGVuZ3RoICAgICAgICAgICAgICAgIDE4CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAg MQogIGJjZFVTQiAgICAgICAgICAgICAgIDMuMDAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5 IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAgICAgICAgIDAgVW51c2VkCiAgYkRldmljZVByb3RvY29s ICAgICAgICAgMyAKICBiTWF4UGFja2V0U2l6ZTAgICAgICAgICA5CiAgaWRWZW5kb3IgICAgICAg ICAgIDB4MDAwMCAKICBpZFByb2R1Y3QgICAgICAgICAgMHgwMDAwIAogIGJjZERldmljZSAgICAg ICAgICAgIDEuMDAKICBpTWFudWZhY3R1cmVyICAgICAgICAgICAxICgweDFiNmYpCiAgaVByb2R1 Y3QgICAgICAgICAgICAgICAgMiBYSENJIHJvb3QgSFVCCiAgaVNlcmlhbCAgICAgICAgICAgICAg ICAgMCAKICBiTnVtQ29uZmlndXJhdGlvbnMgICAgICAxCiAgQ29uZmlndXJhdGlvbiBEZXNjcmlw dG9yOgogICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgYkRlc2NyaXB0b3JUeXBlICAg ICAgICAgMgogICAgd1RvdGFsTGVuZ3RoICAgICAgICAgICAzMQogICAgYk51bUludGVyZmFjZXMg ICAgICAgICAgMQogICAgYkNvbmZpZ3VyYXRpb25WYWx1ZSAgICAgMQogICAgaUNvbmZpZ3VyYXRp b24gICAgICAgICAgMCAKICAgIGJtQXR0cmlidXRlcyAgICAgICAgIDB4NDAKICAgICAgKE1pc3Np bmcgbXVzdC1iZS1zZXQgYml0ISkKICAgICAgU2VsZiBQb3dlcmVkCiAgICBNYXhQb3dlciAgICAg ICAgICAgICAgICAwbUEKICAgIEludGVyZmFjZSBEZXNjcmlwdG9yOgogICAgICBiTGVuZ3RoICAg ICAgICAgICAgICAgICA5CiAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDQKICAgICAgYklu dGVyZmFjZU51bWJlciAgICAgICAgMAogICAgICBiQWx0ZXJuYXRlU2V0dGluZyAgICAgICAwCiAg ICAgIGJOdW1FbmRwb2ludHMgICAgICAgICAgIDEKICAgICAgYkludGVyZmFjZUNsYXNzICAgICAg ICAgOSBIdWIKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAgMCBVbnVzZWQKICAgICAgYklu dGVyZmFjZVByb3RvY29sICAgICAgMCBGdWxsIHNwZWVkIChvciByb290KSBodWIKICAgICAgaUlu dGVyZmFjZSAgICAgICAgICAgICAgMCAKICAgICAgRW5kcG9pbnQgRGVzY3JpcHRvcjoKICAgICAg ICBiTGVuZ3RoICAgICAgICAgICAgICAgICA3CiAgICAgICAgYkRlc2NyaXB0b3JUeXBlICAgICAg ICAgNQogICAgICAgIGJFbmRwb2ludEFkZHJlc3MgICAgIDB4ODEgIEVQIDEgSU4KICAgICAgICBi bUF0dHJpYnV0ZXMgICAgICAgICAgICAzCiAgICAgICAgICBUcmFuc2ZlciBUeXBlICAgICAgICAg ICAgSW50ZXJydXB0CiAgICAgICAgICBTeW5jaCBUeXBlICAgICAgICAgICAgICAgTm9uZQogICAg ICAgICAgVXNhZ2UgVHlwZSAgICAgICAgICAgICAgIERhdGEKICAgICAgICB3TWF4UGFja2V0U2l6 ZSAgICAgMHgwMDAyICAxeCAyIGJ5dGVzCiAgICAgICAgYkludGVydmFsICAgICAgICAgICAgIDI1 NQogICAgICAgIGJNYXhCdXJzdCAgICAgICAgICAgICAgIDEKSHViIERlc2NyaXB0b3I6CiAgYkxl bmd0aCAgICAgICAgICAgICAgNDIKICBiRGVzY3JpcHRvclR5cGUgICAgICA0MgogIG5OYnJQb3J0 cyAgICAgICAgICAgICA0CiAgd0h1YkNoYXJhY3RlcmlzdGljIDB4MDAwOQogICAgUGVyLXBvcnQg cG93ZXIgc3dpdGNoaW5nCiAgICBQZXItcG9ydCBvdmVyY3VycmVudCBwcm90ZWN0aW9uCiAgYlB3 ck9uMlB3ckdvb2QgICAgICAgMTAgKiAyIG1pbGxpIHNlY29uZHMKICBiSHViQ29udHJDdXJyZW50 ICAgICAgMCBtaWxsaSBBbXBlcmUKICBiSHViRGVjTGF0ICAgICAgICAgIDAuMCBtaWNybyBzZWNv bmRzCiAgd0h1YkRlbGF5ICAgICAgICAgICAgIDAgbmFubyBzZWNvbmRzCiAgRGV2aWNlUmVtb3Zh YmxlICAgIDB4MDAKIEh1YiBQb3J0IFN0YXR1czoKICAgUG9ydCAxOiAwMDAwLjA2YTAgVW5rbm93 biBTcGVlZCBwb3dlciBSeC5EZXRlY3QKICAgUG9ydCAyOiAwMDAwLjA2YTAgVW5rbm93biBTcGVl ZCBwb3dlciBSeC5EZXRlY3QKICAgUG9ydCAzOiAwMDAwLjA2YTAgVW5rbm93biBTcGVlZCBwb3dl ciBSeC5EZXRlY3QKICAgUG9ydCA0OiAwMDAwLjA2YTAgVW5rbm93biBTcGVlZCBwb3dlciBSeC5E ZXRlY3QKRGV2aWNlIFN0YXR1czogICAgIDB4MDAwMQogIFNlbGYgUG93ZXJlZApCaW5hcnkgT2Jq ZWN0IFN0b3JlIERlc2NyaXB0b3I6CiAgYkxlbmd0aCAgICAgICAgICAgICAgICAgNQogIGJEZXNj cmlwdG9yVHlwZSAgICAgICAgMTUKICB3VG90YWxMZW5ndGggICAgICAgICAgIDI3CiAgYk51bURl dmljZUNhcHMgICAgICAgICAgMwogIFVTQiAyLjAgRXh0ZW5zaW9uIERldmljZSBDYXBhYmlsaXR5 OgogICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgNwogICAgYkRlc2NyaXB0b3JUeXBlICAgICAg ICAgMQogICAgYkRldkNhcGFiaWxpdHlUeXBlICAgICAgMgogICAgYm1BdHRyaWJ1dGVzICAgMHgw MDAwMDAwMgogICAgICBMaW5rIFBvd2VyIE1hbmFnZW1lbnQgKExQTSkgU3VwcG9ydGVkCiAgU3Vw ZXJTcGVlZCBVU0IgRGV2aWNlIENhcGFiaWxpdHk6CiAgICBiTGVuZ3RoICAgICAgICAgICAgICAg IDEwCiAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgIDE2CiAgICBiRGV2Q2FwYWJpbGl0eVR5cGUg ICAgICAzCiAgICBibUF0dHJpYnV0ZXMgICAgICAgICAweDAwCiAgICAgIExhdGVuY3kgVG9sZXJh bmNlIE1lc3NhZ2VzIChMVE0pIFN1cHBvcnRlZAogICAgd1NwZWVkc1N1cHBvcnRlZCAgIDB4MDAw YwogICAgICBEZXZpY2UgY2FuIG9wZXJhdGUgYXQgSGlnaCBTcGVlZCAoNDgwTWJwcykKICAgICAg RGV2aWNlIGNhbiBvcGVyYXRlIGF0IFN1cGVyU3BlZWQgKDVHYnBzKQogICAgYkZ1bmN0aW9uYWxp dHlTdXBwb3J0ICAgMAogICAgICBMb3dlc3QgZnVsbHktZnVuY3Rpb25hbCBkZXZpY2Ugc3BlZWQg aXMgTG93IFNwZWVkICgxTWJwcykKICAgIGJVMURldkV4aXRMYXQgICAgICAgICAgIDggbWljcm8g c2Vjb25kcwogICAgYlUyRGV2RXhpdExhdCAgICAgICA2NTI4MCBtaWNybyBzZWNvbmRzCiAgQmFk IENvbnRhaW5lciBJRCBEZXZpY2UgQ2FwYWJpbGl0eSBkZXNjcmlwdG9yLgoKQnVzIC9kZXYvdXNi IERldmljZSAvZGV2L3VnZW42LjE6IElEIDAwMDA6MDAwMCAgCkRldmljZSBEZXNjcmlwdG9yOgog IGJMZW5ndGggICAgICAgICAgICAgICAgMTgKICBiRGVzY3JpcHRvclR5cGUgICAgICAgICAxCiAg YmNkVVNCICAgICAgICAgICAgICAgMS4wMAogIGJEZXZpY2VDbGFzcyAgICAgICAgICAgIDkgSHVi CiAgYkRldmljZVN1YkNsYXNzICAgICAgICAgMCBVbnVzZWQKICBiRGV2aWNlUHJvdG9jb2wgICAg ICAgICAwIEZ1bGwgc3BlZWQgKG9yIHJvb3QpIGh1YgogIGJNYXhQYWNrZXRTaXplMCAgICAgICAg NjQKICBpZFZlbmRvciAgICAgICAgICAgMHgwMDAwIAogIGlkUHJvZHVjdCAgICAgICAgICAweDAw MDAgCiAgYmNkRGV2aWNlICAgICAgICAgICAgMS4wMAogIGlNYW51ZmFjdHVyZXIgICAgICAgICAg IDEgQVRJCiAgaVByb2R1Y3QgICAgICAgICAgICAgICAgMiBPSENJIHJvb3QgSFVCCiAgaVNlcmlh bCAgICAgICAgICAgICAgICAgMCAKICBiTnVtQ29uZmlndXJhdGlvbnMgICAgICAxCiAgQ29uZmln dXJhdGlvbiBEZXNjcmlwdG9yOgogICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgYkRl c2NyaXB0b3JUeXBlICAgICAgICAgMgogICAgd1RvdGFsTGVuZ3RoICAgICAgICAgICAyNQogICAg Yk51bUludGVyZmFjZXMgICAgICAgICAgMQogICAgYkNvbmZpZ3VyYXRpb25WYWx1ZSAgICAgMQog ICAgaUNvbmZpZ3VyYXRpb24gICAgICAgICAgMCAKICAgIGJtQXR0cmlidXRlcyAgICAgICAgIDB4 NDAKICAgICAgKE1pc3NpbmcgbXVzdC1iZS1zZXQgYml0ISkKICAgICAgU2VsZiBQb3dlcmVkCiAg ICBNYXhQb3dlciAgICAgICAgICAgICAgICAwbUEKICAgIEludGVyZmFjZSBEZXNjcmlwdG9yOgog ICAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA5CiAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAg ICAgIDQKICAgICAgYkludGVyZmFjZU51bWJlciAgICAgICAgMAogICAgICBiQWx0ZXJuYXRlU2V0 dGluZyAgICAgICAwCiAgICAgIGJOdW1FbmRwb2ludHMgICAgICAgICAgIDEKICAgICAgYkludGVy ZmFjZUNsYXNzICAgICAgICAgOSBIdWIKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAgMCBV bnVzZWQKICAgICAgYkludGVyZmFjZVByb3RvY29sICAgICAgMCBGdWxsIHNwZWVkIChvciByb290 KSBodWIKICAgICAgaUludGVyZmFjZSAgICAgICAgICAgICAgMCAKICAgICAgRW5kcG9pbnQgRGVz Y3JpcHRvcjoKICAgICAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA3CiAgICAgICAgYkRlc2Ny aXB0b3JUeXBlICAgICAgICAgNQogICAgICAgIGJFbmRwb2ludEFkZHJlc3MgICAgIDB4ODEgIEVQ IDEgSU4KICAgICAgICBibUF0dHJpYnV0ZXMgICAgICAgICAgICAzCiAgICAgICAgICBUcmFuc2Zl ciBUeXBlICAgICAgICAgICAgSW50ZXJydXB0CiAgICAgICAgICBTeW5jaCBUeXBlICAgICAgICAg ICAgICAgTm9uZQogICAgICAgICAgVXNhZ2UgVHlwZSAgICAgICAgICAgICAgIERhdGEKICAgICAg ICB3TWF4UGFja2V0U2l6ZSAgICAgMHgwMDIwICAxeCAzMiBieXRlcwogICAgICAgIGJJbnRlcnZh bCAgICAgICAgICAgICAyNTUKSHViIERlc2NyaXB0b3I6CiAgYkxlbmd0aCAgICAgICAgICAgICAg IDkKICBiRGVzY3JpcHRvclR5cGUgICAgICA0MQogIG5OYnJQb3J0cyAgICAgICAgICAgICA0CiAg d0h1YkNoYXJhY3RlcmlzdGljIDB4MDAwMAogICAgR2FuZ2VkIHBvd2VyIHN3aXRjaGluZwogICAg R2FuZ2VkIG92ZXJjdXJyZW50IHByb3RlY3Rpb24KICBiUHdyT24yUHdyR29vZCAgICAgICAgMiAq IDIgbWlsbGkgc2Vjb25kcwogIGJIdWJDb250ckN1cnJlbnQgICAgICAwIG1pbGxpIEFtcGVyZQog IERldmljZVJlbW92YWJsZSAgICAweDAwCiAgUG9ydFB3ckN0cmxNYXNrICAgIDB4MDAKIEh1YiBQ b3J0IFN0YXR1czoKICAgUG9ydCAxOiAwMDAwLjAxMDAgcG93ZXIKICAgUG9ydCAyOiAwMDAwLjAx MDAgcG93ZXIKICAgUG9ydCAzOiAwMDAwLjAxMDAgcG93ZXIKICAgUG9ydCA0OiAwMDAwLjAxMDAg cG93ZXIKRGV2aWNlIFN0YXR1czogICAgIDB4MDAwMQogIFNlbGYgUG93ZXJlZAoKQnVzIC9kZXYv dXNiIERldmljZSAvZGV2L3VnZW4xLjE6IElEIDAwMDA6MDAwMCAgCkRldmljZSBEZXNjcmlwdG9y OgogIGJMZW5ndGggICAgICAgICAgICAgICAgMTgKICBiRGVzY3JpcHRvclR5cGUgICAgICAgICAx CiAgYmNkVVNCICAgICAgICAgICAgICAgMi4wMAogIGJEZXZpY2VDbGFzcyAgICAgICAgICAgIDkg SHViCiAgYkRldmljZVN1YkNsYXNzICAgICAgICAgMCBVbnVzZWQKICBiRGV2aWNlUHJvdG9jb2wg ICAgICAgICAxIFNpbmdsZSBUVAogIGJNYXhQYWNrZXRTaXplMCAgICAgICAgNjQKICBpZFZlbmRv ciAgICAgICAgICAgMHgwMDAwIAogIGlkUHJvZHVjdCAgICAgICAgICAweDAwMDAgCiAgYmNkRGV2 aWNlICAgICAgICAgICAgMS4wMAogIGlNYW51ZmFjdHVyZXIgICAgICAgICAgIDEgQVRJCiAgaVBy b2R1Y3QgICAgICAgICAgICAgICAgMiBFSENJIHJvb3QgSFVCCiAgaVNlcmlhbCAgICAgICAgICAg ICAgICAgMCAKICBiTnVtQ29uZmlndXJhdGlvbnMgICAgICAxCiAgQ29uZmlndXJhdGlvbiBEZXNj cmlwdG9yOgogICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgYkRlc2NyaXB0b3JUeXBl ICAgICAgICAgMgogICAgd1RvdGFsTGVuZ3RoICAgICAgICAgICAyNQogICAgYk51bUludGVyZmFj ZXMgICAgICAgICAgMQogICAgYkNvbmZpZ3VyYXRpb25WYWx1ZSAgICAgMQogICAgaUNvbmZpZ3Vy YXRpb24gICAgICAgICAgMCAKICAgIGJtQXR0cmlidXRlcyAgICAgICAgIDB4NDAKICAgICAgKE1p c3NpbmcgbXVzdC1iZS1zZXQgYml0ISkKICAgICAgU2VsZiBQb3dlcmVkCiAgICBNYXhQb3dlciAg ICAgICAgICAgICAgICAwbUEKICAgIEludGVyZmFjZSBEZXNjcmlwdG9yOgogICAgICBiTGVuZ3Ro ICAgICAgICAgICAgICAgICA5CiAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDQKICAgICAg YkludGVyZmFjZU51bWJlciAgICAgICAgMAogICAgICBiQWx0ZXJuYXRlU2V0dGluZyAgICAgICAw CiAgICAgIGJOdW1FbmRwb2ludHMgICAgICAgICAgIDEKICAgICAgYkludGVyZmFjZUNsYXNzICAg ICAgICAgOSBIdWIKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAgMCBVbnVzZWQKICAgICAg YkludGVyZmFjZVByb3RvY29sICAgICAgMCBGdWxsIHNwZWVkIChvciByb290KSBodWIKICAgICAg aUludGVyZmFjZSAgICAgICAgICAgICAgMCAKICAgICAgRW5kcG9pbnQgRGVzY3JpcHRvcjoKICAg ICAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA3CiAgICAgICAgYkRlc2NyaXB0b3JUeXBlICAg ICAgICAgNQogICAgICAgIGJFbmRwb2ludEFkZHJlc3MgICAgIDB4ODEgIEVQIDEgSU4KICAgICAg ICBibUF0dHJpYnV0ZXMgICAgICAgICAgICAzCiAgICAgICAgICBUcmFuc2ZlciBUeXBlICAgICAg ICAgICAgSW50ZXJydXB0CiAgICAgICAgICBTeW5jaCBUeXBlICAgICAgICAgICAgICAgTm9uZQog ICAgICAgICAgVXNhZ2UgVHlwZSAgICAgICAgICAgICAgIERhdGEKICAgICAgICB3TWF4UGFja2V0 U2l6ZSAgICAgMHgwMDA4ICAxeCA4IGJ5dGVzCiAgICAgICAgYkludGVydmFsICAgICAgICAgICAg IDI1NQpIdWIgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAgICAgICAgICAgOQogIGJEZXNjcmlw dG9yVHlwZSAgICAgIDQxCiAgbk5iclBvcnRzICAgICAgICAgICAgIDUKICB3SHViQ2hhcmFjdGVy aXN0aWMgMHgwMDAyCiAgICBObyBwb3dlciBzd2l0Y2hpbmcgKHVzYiAxLjApCiAgICBHYW5nZWQg b3ZlcmN1cnJlbnQgcHJvdGVjdGlvbgogICAgVFQgdGhpbmsgdGltZSA4IEZTIGJpdHMKICBiUHdy T24yUHdyR29vZCAgICAgIDIwMCAqIDIgbWlsbGkgc2Vjb25kcwogIGJIdWJDb250ckN1cnJlbnQg ICAgICAwIG1pbGxpIEFtcGVyZQogIERldmljZVJlbW92YWJsZSAgICAweDAwCiAgUG9ydFB3ckN0 cmxNYXNrICAgIDB4MDAKIEh1YiBQb3J0IFN0YXR1czoKICAgUG9ydCAxOiAwMDAwLjA1MDAgaGln aHNwZWVkIHBvd2VyCiAgIFBvcnQgMjogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgogICBQb3J0 IDM6IDAwMDAuMDUwMCBoaWdoc3BlZWQgcG93ZXIKICAgUG9ydCA0OiAwMDAwLjA1MDAgaGlnaHNw ZWVkIHBvd2VyCiAgIFBvcnQgNTogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgpEZXZpY2UgUXVh bGlmaWVyIChmb3Igb3RoZXIgZGV2aWNlIHNwZWVkKToKICBiTGVuZ3RoICAgICAgICAgICAgICAg IDEwCiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgNgogIGJjZFVTQiAgICAgICAgICAgICAgIDIu MDAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAgICAg ICAgIDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAgICAgICAgMCBGdWxsIHNwZWVkIChvciBy b290KSBodWIKICBiTWF4UGFja2V0U2l6ZTAgICAgICAgICAwCiAgYk51bUNvbmZpZ3VyYXRpb25z ICAgICAgMApEZXZpY2UgU3RhdHVzOiAgICAgMHgwMDAxCiAgU2VsZiBQb3dlcmVkCgpCdXMgL2Rl di91c2IgRGV2aWNlIC9kZXYvdWdlbjIuMTogSUQgMDAwMDowMDAwICAKRGV2aWNlIERlc2NyaXB0 b3I6CiAgYkxlbmd0aCAgICAgICAgICAgICAgICAxOAogIGJEZXNjcmlwdG9yVHlwZSAgICAgICAg IDEKICBiY2RVU0IgICAgICAgICAgICAgICAxLjAwCiAgYkRldmljZUNsYXNzICAgICAgICAgICAg OSBIdWIKICBiRGV2aWNlU3ViQ2xhc3MgICAgICAgICAwIFVudXNlZAogIGJEZXZpY2VQcm90b2Nv bCAgICAgICAgIDAgRnVsbCBzcGVlZCAob3Igcm9vdCkgaHViCiAgYk1heFBhY2tldFNpemUwICAg ICAgICA2NAogIGlkVmVuZG9yICAgICAgICAgICAweDAwMDAgCiAgaWRQcm9kdWN0ICAgICAgICAg IDB4MDAwMCAKICBiY2REZXZpY2UgICAgICAgICAgICAxLjAwCiAgaU1hbnVmYWN0dXJlciAgICAg ICAgICAgMSBBVEkKICBpUHJvZHVjdCAgICAgICAgICAgICAgICAyIE9IQ0kgcm9vdCBIVUIKICBp U2VyaWFsICAgICAgICAgICAgICAgICAwIAogIGJOdW1Db25maWd1cmF0aW9ucyAgICAgIDEKICBD b25maWd1cmF0aW9uIERlc2NyaXB0b3I6CiAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA5CiAg ICBiRGVzY3JpcHRvclR5cGUgICAgICAgICAyCiAgICB3VG90YWxMZW5ndGggICAgICAgICAgIDI1 CiAgICBiTnVtSW50ZXJmYWNlcyAgICAgICAgICAxCiAgICBiQ29uZmlndXJhdGlvblZhbHVlICAg ICAxCiAgICBpQ29uZmlndXJhdGlvbiAgICAgICAgICAwIAogICAgYm1BdHRyaWJ1dGVzICAgICAg ICAgMHg0MAogICAgICAoTWlzc2luZyBtdXN0LWJlLXNldCBiaXQhKQogICAgICBTZWxmIFBvd2Vy ZWQKICAgIE1heFBvd2VyICAgICAgICAgICAgICAgIDBtQQogICAgSW50ZXJmYWNlIERlc2NyaXB0 b3I6CiAgICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDkKICAgICAgYkRlc2NyaXB0b3JUeXBl ICAgICAgICAgNAogICAgICBiSW50ZXJmYWNlTnVtYmVyICAgICAgICAwCiAgICAgIGJBbHRlcm5h dGVTZXR0aW5nICAgICAgIDAKICAgICAgYk51bUVuZHBvaW50cyAgICAgICAgICAgMQogICAgICBi SW50ZXJmYWNlQ2xhc3MgICAgICAgICA5IEh1YgogICAgICBiSW50ZXJmYWNlU3ViQ2xhc3MgICAg ICAwIFVudXNlZAogICAgICBiSW50ZXJmYWNlUHJvdG9jb2wgICAgICAwIEZ1bGwgc3BlZWQgKG9y IHJvb3QpIGh1YgogICAgICBpSW50ZXJmYWNlICAgICAgICAgICAgICAwIAogICAgICBFbmRwb2lu dCBEZXNjcmlwdG9yOgogICAgICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDcKICAgICAgICBi RGVzY3JpcHRvclR5cGUgICAgICAgICA1CiAgICAgICAgYkVuZHBvaW50QWRkcmVzcyAgICAgMHg4 MSAgRVAgMSBJTgogICAgICAgIGJtQXR0cmlidXRlcyAgICAgICAgICAgIDMKICAgICAgICAgIFRy YW5zZmVyIFR5cGUgICAgICAgICAgICBJbnRlcnJ1cHQKICAgICAgICAgIFN5bmNoIFR5cGUgICAg ICAgICAgICAgICBOb25lCiAgICAgICAgICBVc2FnZSBUeXBlICAgICAgICAgICAgICAgRGF0YQog ICAgICAgIHdNYXhQYWNrZXRTaXplICAgICAweDAwMjAgIDF4IDMyIGJ5dGVzCiAgICAgICAgYklu dGVydmFsICAgICAgICAgICAgIDI1NQpIdWIgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAgICAg ICAgICAgOQogIGJEZXNjcmlwdG9yVHlwZSAgICAgIDQxCiAgbk5iclBvcnRzICAgICAgICAgICAg IDUKICB3SHViQ2hhcmFjdGVyaXN0aWMgMHgwMDAwCiAgICBHYW5nZWQgcG93ZXIgc3dpdGNoaW5n CiAgICBHYW5nZWQgb3ZlcmN1cnJlbnQgcHJvdGVjdGlvbgogIGJQd3JPbjJQd3JHb29kICAgICAg ICAyICogMiBtaWxsaSBzZWNvbmRzCiAgYkh1YkNvbnRyQ3VycmVudCAgICAgIDAgbWlsbGkgQW1w ZXJlCiAgRGV2aWNlUmVtb3ZhYmxlICAgIDB4MDAKICBQb3J0UHdyQ3RybE1hc2sgICAgMHgwMAog SHViIFBvcnQgU3RhdHVzOgogICBQb3J0IDE6IDAwMDAuMDEwMCBwb3dlcgogICBQb3J0IDI6IDAw MDAuMDEwMCBwb3dlcgogICBQb3J0IDM6IDAwMDAuMDEwMCBwb3dlcgogICBQb3J0IDQ6IDAwMDAu MDMwMyBsb3dzcGVlZCBwb3dlciBlbmFibGUgY29ubmVjdAogICBQb3J0IDU6IDAwMDAuMDMwMyBs b3dzcGVlZCBwb3dlciBlbmFibGUgY29ubmVjdApEZXZpY2UgU3RhdHVzOiAgICAgMHgwMDAxCiAg U2VsZiBQb3dlcmVkCg== ------==--bound.487476.omv3tj5zwvd3plqc.iva.yp-c.yandex.net-- From nobody Mon Feb 26 10:21:04 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TjxTy70WDz5CqS0; Mon, 26 Feb 2024 10:21:10 +0000 (UTC) (envelope-from serguey-grigoriev@yandex.ru) Received: from forward502a.mail.yandex.net (forward502a.mail.yandex.net [IPv6:2a02:6b8:c0e:500:1:45:d181:d502]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TjxTx4qb4z4wgh; Mon, 26 Feb 2024 10:21:09 +0000 (UTC) (envelope-from serguey-grigoriev@yandex.ru) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yandex.ru header.s=mail header.b=pgVFKTPJ; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (mx1.freebsd.org: domain of serguey-grigoriev@yandex.ru designates 2a02:6b8:c0e:500:1:45:d181:d502 as permitted sender) smtp.mailfrom=serguey-grigoriev@yandex.ru Received: from mail-nwsmtp-mxback-production-main-90.iva.yp-c.yandex.net (mail-nwsmtp-mxback-production-main-90.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:2da1:0:640:d211:0]) by forward502a.mail.yandex.net (Yandex) with ESMTPS id 9ED1A60BDD; Mon, 26 Feb 2024 13:21:05 +0300 (MSK) Received: from mail.yandex.ru (2a02:6b8:c0c:3b18:0:640:873a:0 [2a02:6b8:c0c:3b18:0:640:873a:0]) by mail-nwsmtp-mxback-production-main-90.iva.yp-c.yandex.net (mxback/Yandex) with HTTP id 5LUH4D0M6Gk0-fYZGIFpI; Mon, 26 Feb 2024 13:21:05 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1708942865; bh=klzeLPyysPIhk2x5BxLj6xPisi2UnUtW05iCIIQbQ3k=; h=Message-Id:References:Date:Cc:Subject:In-Reply-To:To:From; b=pgVFKTPJNpsBeY9G5sp8z06cZlWiofflSnfKJB+ZVXbxbDMOsxOG6Djz0knZnkC7o f+TjI+qzg1NDhbAiP2FIkCp8rHjIAnRfk4nomq7Q9arG7gP73M52YOXGxqBMy4IMyF 98zAabkWiyDGdWMjQrIw6B34EduGh+MJaNfGiwJw= Received: by omv3tj5zwvd3plqc.iva.yp-c.yandex.net with HTTP; Mon, 26 Feb 2024 13:21:04 +0300 From: S.N.Grigoriev To: FreeBSD Stable Cc: stable@freebsd.org In-Reply-To: <5a581a59-94fc-4476-97c4-4afac65c27bf@app.fastmail.com> References: <5813111708783942@wckjnuw7tehz7zgt.sas.yp-c.yandex.net> <3c8667a2-770c-4c35-9b33-572f43d9f1ea@app.fastmail.com> <4609701708855679@vpfns56j3qv6ec2k.vla.yp-c.yandex.net> <511954d9-74b4-45c3-b16e-342ad29751f7@app.fastmail.com> <478521708877670@z7oys7c4urhdnodv.sas.yp-c.yandex.net> <5a581a59-94fc-4476-97c4-4afac65c27bf@app.fastmail.com> Subject: Re: USB CD drive does not work with 14-Stable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Mon, 26 Feb 2024 13:21:04 +0300 Message-Id: <4874751708942864@omv3tj5zwvd3plqc.iva.yp-c.yandex.net> Content-Type: multipart/mixed; boundary="----==--bound.487476.omv3tj5zwvd3plqc.iva.yp-c.yandex.net" X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.20 / 15.00]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yandex.ru,none]; NEURAL_HAM_SHORT(-0.20)[-0.205]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:c00::/40]; R_DKIM_ALLOW(-0.20)[yandex.ru:s=mail]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[2a02:6b8:c0e:500:1:45:d181:d502:from]; MIME_BASE64_TEXT(0.10)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[yandex.ru]; MIME_TRACE(0.00)[0:+,1:+,2:+]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org,stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; HAS_ATTACHMENT(0.00)[]; DKIM_TRACE(0.00)[yandex.ru:+]; FREEMAIL_ENVFROM(0.00)[yandex.ru]; DWL_DNSWL_NONE(0.00)[yandex.ru:dkim] X-Rspamd-Queue-Id: 4TjxTx4qb4z4wgh ------==--bound.487476.omv3tj5zwvd3plqc.iva.yp-c.yandex.net Content-Transfer-Encoding: 7bit Content-Type: text/plain > On Sun, 25 Feb 2024, at 16:14, S.N.Grigoriev wrote: > >>> Another thing maybe worthwhile trying is to just have that one thing >>> plugged into the usb subsystem. Beforehand, tail -f /var/log/messages >>> in an xterm, unplug everything usb (2&3), plug in the cd drive, >>> observe messages. >> >> I've done it. The same result. > > What is the output of lsusb -v ? > > lsusb output in the file attached. The following error message displayed during execution: can't get debug descriptor: Input/output error. Does it mean that my USB hardware is broken? Regards, Serguey. ------==--bound.487476.omv3tj5zwvd3plqc.iva.yp-c.yandex.net Content-Disposition: attachment; filename="lsusb.txt" Content-Transfer-Encoding: base64 Content-Type: text/plain; name="lsusb.txt" CkJ1cyAvZGV2L3VzYiBEZXZpY2UgL2Rldi91Z2VuMi4zOiBJRCAxYTJjOjBjMjEgQ2hpbmEgUmVz b3VyY2UgU2VtaWNvIENvLiwgTHRkIApEZXZpY2UgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAg ICAgICAgICAgIDE4CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgMQogIGJjZFVTQiAgICAgICAg ICAgICAgIDEuMTAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICAwIChEZWZpbmVkIGF0IEludGVy ZmFjZSBsZXZlbCkKICBiRGV2aWNlU3ViQ2xhc3MgICAgICAgICAwIAogIGJEZXZpY2VQcm90b2Nv bCAgICAgICAgIDAgCiAgYk1heFBhY2tldFNpemUwICAgICAgICAgOAogIGlkVmVuZG9yICAgICAg ICAgICAweDFhMmMgQ2hpbmEgUmVzb3VyY2UgU2VtaWNvIENvLiwgTHRkCiAgaWRQcm9kdWN0ICAg ICAgICAgIDB4MGMyMSAKICBiY2REZXZpY2UgICAgICAgICAgICAxLjEwCiAgaU1hbnVmYWN0dXJl ciAgICAgICAgICAgMSBVU0IKICBpUHJvZHVjdCAgICAgICAgICAgICAgICAyIFVTQiBLZXlib2Fy ZAogIGlTZXJpYWwgICAgICAgICAgICAgICAgIDAgCiAgYk51bUNvbmZpZ3VyYXRpb25zICAgICAg MQogIENvbmZpZ3VyYXRpb24gRGVzY3JpcHRvcjoKICAgIGJMZW5ndGggICAgICAgICAgICAgICAg IDkKICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDIKICAgIHdUb3RhbExlbmd0aCAgICAgICAg ICAgNTkKICAgIGJOdW1JbnRlcmZhY2VzICAgICAgICAgIDIKICAgIGJDb25maWd1cmF0aW9uVmFs dWUgICAgIDEKICAgIGlDb25maWd1cmF0aW9uICAgICAgICAgIDAgCiAgICBibUF0dHJpYnV0ZXMg ICAgICAgICAweGEwCiAgICAgIChCdXMgUG93ZXJlZCkKICAgICAgUmVtb3RlIFdha2V1cAogICAg TWF4UG93ZXIgICAgICAgICAgICAgIDEwMG1BCiAgICBJbnRlcmZhY2UgRGVzY3JpcHRvcjoKICAg ICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgICBiRGVzY3JpcHRvclR5cGUgICAgICAg ICA0CiAgICAgIGJJbnRlcmZhY2VOdW1iZXIgICAgICAgIDAKICAgICAgYkFsdGVybmF0ZVNldHRp bmcgICAgICAgMAogICAgICBiTnVtRW5kcG9pbnRzICAgICAgICAgICAxCiAgICAgIGJJbnRlcmZh Y2VDbGFzcyAgICAgICAgIDMgSHVtYW4gSW50ZXJmYWNlIERldmljZQogICAgICBiSW50ZXJmYWNl U3ViQ2xhc3MgICAgICAxIEJvb3QgSW50ZXJmYWNlIFN1YmNsYXNzCiAgICAgIGJJbnRlcmZhY2VQ cm90b2NvbCAgICAgIDEgS2V5Ym9hcmQKICAgICAgaUludGVyZmFjZSAgICAgICAgICAgICAgMCAK ICAgICAgICBISUQgRGV2aWNlIERlc2NyaXB0b3I6CiAgICAgICAgICBiTGVuZ3RoICAgICAgICAg ICAgICAgICA5CiAgICAgICAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgIDMzCiAgICAgICAgICBi Y2RISUQgICAgICAgICAgICAgICAxLjEwCiAgICAgICAgICBiQ291bnRyeUNvZGUgICAgICAgICAg ICAwIE5vdCBzdXBwb3J0ZWQKICAgICAgICAgIGJOdW1EZXNjcmlwdG9ycyAgICAgICAgIDEKICAg ICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgMzQgUmVwb3J0CiAgICAgICAgICB3RGVzY3Jp cHRvckxlbmd0aCAgICAgIDU0CiAgICAgICAgICBSZXBvcnQgRGVzY3JpcHRvcjogKGxlbmd0aCBp cyA1NCkKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBVc2FnZSBQYWdlLCBkYXRhPSBbIDB4MDEg XSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBHZW5lcmljIERlc2t0b3AgQ29udHJvbHMK ICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSwgZGF0YT0gWyAweDA2IF0gNgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgS2V5Ym9hcmQKICAgICAgICAgICAgSXRlbShNYWluICApOiBD b2xsZWN0aW9uLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBB cHBsaWNhdGlvbgogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2UsIGRhdGE9IFsg MHgwOCBdIDgKICAgICAgICAgICAgICAgICAgICAgICAgICAgIExFRHMKICAgICAgICAgICAgSXRl bShMb2NhbCApOiBVc2FnZSBNaW5pbXVtLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICBOdW1Mb2NrCiAgICAgICAgICAgIEl0ZW0oTG9jYWwgKTogVXNhZ2UgTWF4 aW11bSwgZGF0YT0gWyAweDAzIF0gMwogICAgICAgICAgICAgICAgICAgICAgICAgICAgU2Nyb2xs IExvY2sKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1pbmltdW0sIGRhdGE9IFsg MHgwMCBdIDAKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1heGltdW0sIGRhdGE9 IFsgMHgwMSBdIDEKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQgU2l6ZSwgZGF0YT0g WyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0g WyAweDAzIF0gMwogICAgICAgICAgICBJdGVtKE1haW4gICk6IE91dHB1dCwgZGF0YT0gWyAweDAy IF0gMgogICAgICAgICAgICAgICAgICAgICAgICAgICAgRGF0YSBWYXJpYWJsZSBBYnNvbHV0ZSBO b19XcmFwIExpbmVhcgogICAgICAgICAgICAgICAgICAgICAgICAgICAgUHJlZmVycmVkX1N0YXRl IE5vX051bGxfUG9zaXRpb24gTm9uX1ZvbGF0aWxlIEJpdGZpZWxkCiAgICAgICAgICAgIEl0ZW0o R2xvYmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDUgXSA1CiAgICAgICAgICAgIEl0ZW0o TWFpbiAgKTogT3V0cHV0LCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICBDb25zdGFudCBBcnJheSBBYnNvbHV0ZSBOb19XcmFwIExpbmVhcgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgUHJlZmVycmVkX1N0YXRlIE5vX051bGxfUG9zaXRpb24gTm9uX1ZvbGF0 aWxlIEJpdGZpZWxkCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogVXNhZ2UgUGFnZSwgZGF0YT0g WyAweDA3IF0gNwogICAgICAgICAgICAgICAgICAgICAgICAgICAgS2V5Ym9hcmQKICAgICAgICAg ICAgSXRlbShMb2NhbCApOiBVc2FnZSBNaW5pbXVtLCBkYXRhPSBbIDB4ZTAgXSAyMjQKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIENvbnRyb2wgTGVmdAogICAgICAgICAgICBJdGVtKExvY2Fs ICk6IFVzYWdlIE1heGltdW0sIGRhdGE9IFsgMHhlNyBdIDIzMQogICAgICAgICAgICAgICAgICAg ICAgICAgICAgR1VJIFJpZ2h0CiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50 LCBkYXRhPSBbIDB4MDggXSA4CiAgICAgICAgICAgIEl0ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9 IFsgMHgwMiBdIDIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIERhdGEgVmFyaWFibGUgQWJz b2x1dGUgTm9fV3JhcCBMaW5lYXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFByZWZlcnJl ZF9TdGF0ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9Wb2xhdGlsZSBCaXRmaWVsZAogICAgICAgICAg ICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBTaXplLCBkYXRhPSBbIDB4MDggXSA4CiAgICAgICAgICAg IEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAg IEl0ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIENvbnN0YW50IEFycmF5IEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlvbiBOb25f Vm9sYXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSBNaW5pbXVt LCBkYXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBObyBFdmVudAog ICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlIE1heGltdW0sIGRhdGE9IFsgMHg5MSBdIDE0 NQogICAgICAgICAgICAgICAgICAgICAgICAgICAgTEFORyAyIChIYW5qYSBDb252ZXJzaW9uLCBL b3JlYSkKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1heGltdW0sIGRhdGE9IFsg MHhmZiAweDAwIF0gMjU1CiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBk YXRhPSBbIDB4MDYgXSA2CiAgICAgICAgICAgIEl0ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9IFsg MHgwMCBdIDAKICAgICAgICAgICAgICAgICAgICAgICAgICAgIERhdGEgQXJyYXkgQWJzb2x1dGUg Tm9fV3JhcCBMaW5lYXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFByZWZlcnJlZF9TdGF0 ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9Wb2xhdGlsZSBCaXRmaWVsZAogICAgICAgICAgICBJdGVt KE1haW4gICk6IEVuZCBDb2xsZWN0aW9uLCBkYXRhPW5vbmUKICAgICAgRW5kcG9pbnQgRGVzY3Jp cHRvcjoKICAgICAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA3CiAgICAgICAgYkRlc2NyaXB0 b3JUeXBlICAgICAgICAgNQogICAgICAgIGJFbmRwb2ludEFkZHJlc3MgICAgIDB4ODEgIEVQIDEg SU4KICAgICAgICBibUF0dHJpYnV0ZXMgICAgICAgICAgICAzCiAgICAgICAgICBUcmFuc2ZlciBU eXBlICAgICAgICAgICAgSW50ZXJydXB0CiAgICAgICAgICBTeW5jaCBUeXBlICAgICAgICAgICAg ICAgTm9uZQogICAgICAgICAgVXNhZ2UgVHlwZSAgICAgICAgICAgICAgIERhdGEKICAgICAgICB3 TWF4UGFja2V0U2l6ZSAgICAgMHgwMDA4ICAxeCA4IGJ5dGVzCiAgICAgICAgYkludGVydmFsICAg ICAgICAgICAgICAxMAogICAgSW50ZXJmYWNlIERlc2NyaXB0b3I6CiAgICAgIGJMZW5ndGggICAg ICAgICAgICAgICAgIDkKICAgICAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgNAogICAgICBiSW50 ZXJmYWNlTnVtYmVyICAgICAgICAxCiAgICAgIGJBbHRlcm5hdGVTZXR0aW5nICAgICAgIDAKICAg ICAgYk51bUVuZHBvaW50cyAgICAgICAgICAgMQogICAgICBiSW50ZXJmYWNlQ2xhc3MgICAgICAg ICAzIEh1bWFuIEludGVyZmFjZSBEZXZpY2UKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAg MSBCb290IEludGVyZmFjZSBTdWJjbGFzcwogICAgICBiSW50ZXJmYWNlUHJvdG9jb2wgICAgICAy IE1vdXNlCiAgICAgIGlJbnRlcmZhY2UgICAgICAgICAgICAgIDAgCiAgICAgICAgSElEIERldmlj ZSBEZXNjcmlwdG9yOgogICAgICAgICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgICAg ICAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAzMwogICAgICAgICAgYmNkSElEICAgICAgICAgICAg ICAgMS4xMAogICAgICAgICAgYkNvdW50cnlDb2RlICAgICAgICAgICAgMCBOb3Qgc3VwcG9ydGVk CiAgICAgICAgICBiTnVtRGVzY3JpcHRvcnMgICAgICAgICAxCiAgICAgICAgICBiRGVzY3JpcHRv clR5cGUgICAgICAgIDM0IFJlcG9ydAogICAgICAgICAgd0Rlc2NyaXB0b3JMZW5ndGggICAgIDEw OAogICAgICAgICAgUmVwb3J0IERlc2NyaXB0b3I6IChsZW5ndGggaXMgMTA4KQogICAgICAgICAg ICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2UsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAg ICAgICAgICAgICAgICAgIEdlbmVyaWMgRGVza3RvcCBDb250cm9scwogICAgICAgICAgICBJdGVt KExvY2FsICk6IFVzYWdlLCBkYXRhPSBbIDB4MDIgXSAyCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICBNb3VzZQogICAgICAgICAgICBJdGVtKE1haW4gICk6IENvbGxlY3Rpb24sIGRhdGE9IFsg MHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIEFwcGxpY2F0aW9uCiAgICAgICAg ICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IElELCBkYXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAg IEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIFBvaW50ZXIKICAgICAgICAgICAgSXRlbShNYWluICApOiBDb2xsZWN0aW9uLCBk YXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBQaHlzaWNhbAogICAg ICAgICAgICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2UsIGRhdGE9IFsgMHgwOSBdIDkKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIEJ1dHRvbnMKICAgICAgICAgICAgSXRlbShMb2NhbCApOiBV c2FnZSBNaW5pbXVtLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICBCdXR0b24gMSAoUHJpbWFyeSkKICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSBNYXhp bXVtLCBkYXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBCdXR0b24g MyAoVGVydGlhcnkpCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogTG9naWNhbCBNaW5pbXVtLCBk YXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogTG9naWNhbCBNYXhpbXVt LCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50 LCBkYXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IFNpemUs IGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgSXRlbShNYWluICApOiBJbnB1dCwgZGF0YT0g WyAweDAyIF0gMgogICAgICAgICAgICAgICAgICAgICAgICAgICAgRGF0YSBWYXJpYWJsZSBBYnNv bHV0ZSBOb19XcmFwIExpbmVhcgogICAgICAgICAgICAgICAgICAgICAgICAgICAgUHJlZmVycmVk X1N0YXRlIE5vX051bGxfUG9zaXRpb24gTm9uX1ZvbGF0aWxlIEJpdGZpZWxkCiAgICAgICAgICAg IEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAg IEl0ZW0oR2xvYmFsKTogUmVwb3J0IFNpemUsIGRhdGE9IFsgMHgwNSBdIDUKICAgICAgICAgICAg SXRlbShNYWluICApOiBJbnB1dCwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICAgICAgICAg ICAgICAgICAgQ29uc3RhbnQgQXJyYXkgQWJzb2x1dGUgTm9fV3JhcCBMaW5lYXIKICAgICAgICAg ICAgICAgICAgICAgICAgICAgIFByZWZlcnJlZF9TdGF0ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9W b2xhdGlsZSBCaXRmaWVsZAogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2UsIGRh dGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIEdlbmVyaWMgRGVza3Rv cCBDb250cm9scwogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlLCBkYXRhPSBbIDB4MzAg XSA0OAogICAgICAgICAgICAgICAgICAgICAgICAgICAgRGlyZWN0aW9uLVgKICAgICAgICAgICAg SXRlbShMb2NhbCApOiBVc2FnZSwgZGF0YT0gWyAweDMxIF0gNDkKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIERpcmVjdGlvbi1ZCiAgICAgICAgICAgIEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRh dGE9IFsgMHgzOCBdIDU2CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBXaGVlbAogICAgICAg ICAgICBJdGVtKEdsb2JhbCk6IExvZ2ljYWwgTWluaW11bSwgZGF0YT0gWyAweDgxIF0gMTI5CiAg ICAgICAgICAgIEl0ZW0oR2xvYmFsKTogTG9naWNhbCBNYXhpbXVtLCBkYXRhPSBbIDB4N2YgXSAx MjcKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQgU2l6ZSwgZGF0YT0gWyAweDA4IF0g OAogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0gWyAweDAzIF0g MwogICAgICAgICAgICBJdGVtKE1haW4gICk6IElucHV0LCBkYXRhPSBbIDB4MDYgXSA2CiAgICAg ICAgICAgICAgICAgICAgICAgICAgICBEYXRhIFZhcmlhYmxlIFJlbGF0aXZlIE5vX1dyYXAgTGlu ZWFyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Q b3NpdGlvbiBOb25fVm9sYXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShNYWluICApOiBF bmQgQ29sbGVjdGlvbiwgZGF0YT1ub25lCiAgICAgICAgICAgIEl0ZW0oTWFpbiAgKTogRW5kIENv bGxlY3Rpb24sIGRhdGE9bm9uZQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFVzYWdlIFBhZ2Us IGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIEdlbmVyaWMgRGVz a3RvcCBDb250cm9scwogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlLCBkYXRhPSBbIDB4 ODAgXSAxMjgKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFN5c3RlbSBDb250cm9sCiAgICAg ICAgICAgIEl0ZW0oTWFpbiAgKTogQ29sbGVjdGlvbiwgZGF0YT0gWyAweDAxIF0gMQogICAgICAg ICAgICAgICAgICAgICAgICAgICAgQXBwbGljYXRpb24KICAgICAgICAgICAgSXRlbShHbG9iYWwp OiBSZXBvcnQgSUQsIGRhdGE9IFsgMHgwMiBdIDIKICAgICAgICAgICAgSXRlbShMb2NhbCApOiBV c2FnZSBNaW5pbXVtLCBkYXRhPSBbIDB4ODEgXSAxMjkKICAgICAgICAgICAgICAgICAgICAgICAg ICAgIFN5c3RlbSBQb3dlciBEb3duCiAgICAgICAgICAgIEl0ZW0oTG9jYWwgKTogVXNhZ2UgTWF4 aW11bSwgZGF0YT0gWyAweDgzIF0gMTMxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBTeXN0 ZW0gV2FrZSBVcAogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IExvZ2ljYWwgTWluaW11bSwgZGF0 YT0gWyAweDAwIF0gMAogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IExvZ2ljYWwgTWF4aW11bSwg ZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBTaXplLCBk YXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBk YXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAgIEl0ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9IFsg MHgwMiBdIDIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIERhdGEgVmFyaWFibGUgQWJzb2x1 dGUgTm9fV3JhcCBMaW5lYXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFByZWZlcnJlZF9T dGF0ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9Wb2xhdGlsZSBCaXRmaWVsZAogICAgICAgICAgICBJ dGVtKEdsb2JhbCk6IFJlcG9ydCBTaXplLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgIEl0 ZW0oR2xvYmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDUgXSA1CiAgICAgICAgICAgIEl0 ZW0oTWFpbiAgKTogSW5wdXQsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAg ICAgICAgIENvbnN0YW50IEFycmF5IEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlvbiBOb25fVm9s YXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShNYWluICApOiBFbmQgQ29sbGVjdGlvbiwg ZGF0YT1ub25lCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogVXNhZ2UgUGFnZSwgZGF0YT0gWyAw eDBjIF0gMTIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIENvbnN1bWVyCiAgICAgICAgICAg IEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIENvbnN1bWVyIENvbnRyb2wKICAgICAgICAgICAgSXRlbShNYWluICApOiBDb2xs ZWN0aW9uLCBkYXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBBcHBs aWNhdGlvbgogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBJRCwgZGF0YT0gWyAweDAx IF0gMQogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlIE1pbmltdW0sIGRhdGE9IFsgMHgw MCBdIDAKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFVuYXNzaWduZWQKICAgICAgICAgICAg SXRlbShMb2NhbCApOiBVc2FnZSBNYXhpbXVtLCBkYXRhPSBbIDB4M2MgMHgwMiBdIDU3MgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgQUMgRm9ybWF0CiAgICAgICAgICAgIEl0ZW0oR2xvYmFs KTogTG9naWNhbCBNaW5pbXVtLCBkYXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgIEl0ZW0oR2xv YmFsKTogTG9naWNhbCBNYXhpbXVtLCBkYXRhPSBbIDB4M2MgMHgwMiBdIDU3MgogICAgICAgICAg ICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAg ICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBTaXplLCBkYXRhPSBbIDB4MTAgXSAxNgogICAgICAgICAg ICBJdGVtKE1haW4gICk6IElucHV0LCBkYXRhPSBbIDB4MDAgXSAwCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICBEYXRhIEFycmF5IEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlvbiBOb25fVm9s YXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShNYWluICApOiBFbmQgQ29sbGVjdGlvbiwg ZGF0YT1ub25lCiAgICAgIEVuZHBvaW50IERlc2NyaXB0b3I6CiAgICAgICAgYkxlbmd0aCAgICAg ICAgICAgICAgICAgNwogICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDUKICAgICAgICBi RW5kcG9pbnRBZGRyZXNzICAgICAweDgyICBFUCAyIElOCiAgICAgICAgYm1BdHRyaWJ1dGVzICAg ICAgICAgICAgMwogICAgICAgICAgVHJhbnNmZXIgVHlwZSAgICAgICAgICAgIEludGVycnVwdAog ICAgICAgICAgU3luY2ggVHlwZSAgICAgICAgICAgICAgIE5vbmUKICAgICAgICAgIFVzYWdlIFR5 cGUgICAgICAgICAgICAgICBEYXRhCiAgICAgICAgd01heFBhY2tldFNpemUgICAgIDB4MDAwNSAg MXggNSBieXRlcwogICAgICAgIGJJbnRlcnZhbCAgICAgICAgICAgICAgMTAKRGV2aWNlIFN0YXR1 czogICAgIDB4MDAwMAogIChCdXMgUG93ZXJlZCkKCkJ1cyAvZGV2L3VzYiBEZXZpY2UgL2Rldi91 Z2VuMi4yOiBJRCAwNDVlOjAwNDAgTWljcm9zb2Z0IENvcnAuIFdoZWVsIE1vdXNlIE9wdGljYWwK RGV2aWNlIERlc2NyaXB0b3I6CiAgYkxlbmd0aCAgICAgICAgICAgICAgICAxOAogIGJEZXNjcmlw dG9yVHlwZSAgICAgICAgIDEKICBiY2RVU0IgICAgICAgICAgICAgICAxLjEwCiAgYkRldmljZUNs YXNzICAgICAgICAgICAgMCAoRGVmaW5lZCBhdCBJbnRlcmZhY2UgbGV2ZWwpCiAgYkRldmljZVN1 YkNsYXNzICAgICAgICAgMCAKICBiRGV2aWNlUHJvdG9jb2wgICAgICAgICAwIAogIGJNYXhQYWNr ZXRTaXplMCAgICAgICAgIDgKICBpZFZlbmRvciAgICAgICAgICAgMHgwNDVlIE1pY3Jvc29mdCBD b3JwLgogIGlkUHJvZHVjdCAgICAgICAgICAweDAwNDAgV2hlZWwgTW91c2UgT3B0aWNhbAogIGJj ZERldmljZSAgICAgICAgICAgIDMuMDAKICBpTWFudWZhY3R1cmVyICAgICAgICAgICAxIE1pY3Jv c29mdAogIGlQcm9kdWN0ICAgICAgICAgICAgICAgIDMgTWljcm9zb2Z0IDMtQnV0dG9uIE1vdXNl IHdpdGggSW50ZWxsaUV5ZShUTSkKICBpU2VyaWFsICAgICAgICAgICAgICAgICAwIAogIGJOdW1D b25maWd1cmF0aW9ucyAgICAgIDEKICBDb25maWd1cmF0aW9uIERlc2NyaXB0b3I6CiAgICBiTGVu Z3RoICAgICAgICAgICAgICAgICA5CiAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgICAyCiAgICB3 VG90YWxMZW5ndGggICAgICAgICAgIDM0CiAgICBiTnVtSW50ZXJmYWNlcyAgICAgICAgICAxCiAg ICBiQ29uZmlndXJhdGlvblZhbHVlICAgICAxCiAgICBpQ29uZmlndXJhdGlvbiAgICAgICAgICAw IAogICAgYm1BdHRyaWJ1dGVzICAgICAgICAgMHhhMAogICAgICAoQnVzIFBvd2VyZWQpCiAgICAg IFJlbW90ZSBXYWtldXAKICAgIE1heFBvd2VyICAgICAgICAgICAgICAxMDBtQQogICAgSW50ZXJm YWNlIERlc2NyaXB0b3I6CiAgICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDkKICAgICAgYkRl c2NyaXB0b3JUeXBlICAgICAgICAgNAogICAgICBiSW50ZXJmYWNlTnVtYmVyICAgICAgICAwCiAg ICAgIGJBbHRlcm5hdGVTZXR0aW5nICAgICAgIDAKICAgICAgYk51bUVuZHBvaW50cyAgICAgICAg ICAgMQogICAgICBiSW50ZXJmYWNlQ2xhc3MgICAgICAgICAzIEh1bWFuIEludGVyZmFjZSBEZXZp Y2UKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAgMSBCb290IEludGVyZmFjZSBTdWJjbGFz cwogICAgICBiSW50ZXJmYWNlUHJvdG9jb2wgICAgICAyIE1vdXNlCiAgICAgIGlJbnRlcmZhY2Ug ICAgICAgICAgICAgIDAgCiAgICAgICAgSElEIERldmljZSBEZXNjcmlwdG9yOgogICAgICAgICAg Ykxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgICAgICAgYkRlc2NyaXB0b3JUeXBlICAgICAg ICAzMwogICAgICAgICAgYmNkSElEICAgICAgICAgICAgICAgMS4xMAogICAgICAgICAgYkNvdW50 cnlDb2RlICAgICAgICAgICAgMCBOb3Qgc3VwcG9ydGVkCiAgICAgICAgICBiTnVtRGVzY3JpcHRv cnMgICAgICAgICAxCiAgICAgICAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgIDM0IFJlcG9ydAog ICAgICAgICAgd0Rlc2NyaXB0b3JMZW5ndGggICAgICA3MgogICAgICAgICAgUmVwb3J0IERlc2Ny aXB0b3I6IChsZW5ndGggaXMgNzIpCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogVXNhZ2UgUGFn ZSwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgR2VuZXJpYyBE ZXNrdG9wIENvbnRyb2xzCiAgICAgICAgICAgIEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRhdGE9IFsg MHgwMiBdIDIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIE1vdXNlCiAgICAgICAgICAgIEl0 ZW0oTWFpbiAgKTogQ29sbGVjdGlvbiwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICAgICAg ICAgICAgICAgICAgQXBwbGljYXRpb24KICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSwg ZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgUG9pbnRlcgogICAg ICAgICAgICBJdGVtKE1haW4gICk6IENvbGxlY3Rpb24sIGRhdGE9IFsgMHgwMCBdIDAKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIFBoeXNpY2FsCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTog VXNhZ2UgUGFnZSwgZGF0YT0gWyAweDA5IF0gOQogICAgICAgICAgICAgICAgICAgICAgICAgICAg QnV0dG9ucwogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlIE1pbmltdW0sIGRhdGE9IFsg MHgwMSBdIDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIEJ1dHRvbiAxIChQcmltYXJ5KQog ICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlIE1heGltdW0sIGRhdGE9IFsgMHgwMyBdIDMK ICAgICAgICAgICAgICAgICAgICAgICAgICAgIEJ1dHRvbiAzIChUZXJ0aWFyeSkKICAgICAgICAg ICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1pbmltdW0sIGRhdGE9IFsgMHgwMCBdIDAKICAgICAg ICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1heGltdW0sIGRhdGE9IFsgMHgwMSBdIDEKICAg ICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQgU2l6ZSwgZGF0YT0gWyAweDAxIF0gMQogICAg ICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0gWyAweDAzIF0gMwogICAg ICAgICAgICBJdGVtKE1haW4gICk6IElucHV0LCBkYXRhPSBbIDB4MDIgXSAyCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICBEYXRhIFZhcmlhYmxlIEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlv biBOb25fVm9sYXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQg U2l6ZSwgZGF0YT0gWyAweDA1IF0gNQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBD b3VudCwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKE1haW4gICk6IElucHV0LCBk YXRhPSBbIDB4MDEgXSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBDb25zdGFudCBBcnJh eSBBYnNvbHV0ZSBOb19XcmFwIExpbmVhcgogICAgICAgICAgICAgICAgICAgICAgICAgICAgUHJl ZmVycmVkX1N0YXRlIE5vX051bGxfUG9zaXRpb24gTm9uX1ZvbGF0aWxlIEJpdGZpZWxkCiAgICAg ICAgICAgIEl0ZW0oR2xvYmFsKTogVXNhZ2UgUGFnZSwgZGF0YT0gWyAweDAxIF0gMQogICAgICAg ICAgICAgICAgICAgICAgICAgICAgR2VuZXJpYyBEZXNrdG9wIENvbnRyb2xzCiAgICAgICAgICAg IEl0ZW0oTG9jYWwgKTogVXNhZ2UsIGRhdGE9IFsgMHgzMCBdIDQ4CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICBEaXJlY3Rpb24tWAogICAgICAgICAgICBJdGVtKExvY2FsICk6IFVzYWdlLCBk YXRhPSBbIDB4MzEgXSA0OQogICAgICAgICAgICAgICAgICAgICAgICAgICAgRGlyZWN0aW9uLVkK ICAgICAgICAgICAgSXRlbShMb2NhbCApOiBVc2FnZSwgZGF0YT0gWyAweDM4IF0gNTYKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIFdoZWVsCiAgICAgICAgICAgIEl0ZW0oR2xvYmFsKTogTG9n aWNhbCBNaW5pbXVtLCBkYXRhPSBbIDB4ODEgXSAxMjkKICAgICAgICAgICAgSXRlbShHbG9iYWwp OiBMb2dpY2FsIE1heGltdW0sIGRhdGE9IFsgMHg3ZiBdIDEyNwogICAgICAgICAgICBJdGVtKEds b2JhbCk6IFJlcG9ydCBTaXplLCBkYXRhPSBbIDB4MDggXSA4CiAgICAgICAgICAgIEl0ZW0oR2xv YmFsKTogUmVwb3J0IENvdW50LCBkYXRhPSBbIDB4MDMgXSAzCiAgICAgICAgICAgIEl0ZW0oTWFp biAgKTogSW5wdXQsIGRhdGE9IFsgMHgwNiBdIDYKICAgICAgICAgICAgICAgICAgICAgICAgICAg IERhdGEgVmFyaWFibGUgUmVsYXRpdmUgTm9fV3JhcCBMaW5lYXIKICAgICAgICAgICAgICAgICAg ICAgICAgICAgIFByZWZlcnJlZF9TdGF0ZSBOb19OdWxsX1Bvc2l0aW9uIE5vbl9Wb2xhdGlsZSBC aXRmaWVsZAogICAgICAgICAgICBJdGVtKE1haW4gICk6IEVuZCBDb2xsZWN0aW9uLCBkYXRhPW5v bmUKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBVc2FnZSBQYWdlLCBkYXRhPSBbIDB4ZmYgXSAy NTUKICAgICAgICAgICAgICAgICAgICAgICAgICAgIFZlbmRvciBTcGVjaWZpYwogICAgICAgICAg ICBJdGVtKExvY2FsICk6IFVzYWdlLCBkYXRhPSBbIDB4MDIgXSAyCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAobnVsbCkKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1pbmlt dW0sIGRhdGE9IFsgMHgwMCBdIDAKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBMb2dpY2FsIE1h eGltdW0sIGRhdGE9IFsgMHgwMSBdIDEKICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQg U2l6ZSwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBD b3VudCwgZGF0YT0gWyAweDAxIF0gMQogICAgICAgICAgICBJdGVtKE1haW4gICk6IEZlYXR1cmUs IGRhdGE9IFsgMHgyMiBdIDM0CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBEYXRhIFZhcmlh YmxlIEFic29sdXRlIE5vX1dyYXAgTGluZWFyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBO b19QcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Qb3NpdGlvbiBOb25fVm9sYXRpbGUgQml0ZmllbGQK ICAgICAgICAgICAgSXRlbShHbG9iYWwpOiBSZXBvcnQgU2l6ZSwgZGF0YT0gWyAweDA3IF0gNwog ICAgICAgICAgICBJdGVtKEdsb2JhbCk6IFJlcG9ydCBDb3VudCwgZGF0YT0gWyAweDAxIF0gMQog ICAgICAgICAgICBJdGVtKE1haW4gICk6IEZlYXR1cmUsIGRhdGE9IFsgMHgwMSBdIDEKICAgICAg ICAgICAgICAgICAgICAgICAgICAgIENvbnN0YW50IEFycmF5IEFic29sdXRlIE5vX1dyYXAgTGlu ZWFyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBQcmVmZXJyZWRfU3RhdGUgTm9fTnVsbF9Q b3NpdGlvbiBOb25fVm9sYXRpbGUgQml0ZmllbGQKICAgICAgICAgICAgSXRlbShNYWluICApOiBF bmQgQ29sbGVjdGlvbiwgZGF0YT1ub25lCiAgICAgIEVuZHBvaW50IERlc2NyaXB0b3I6CiAgICAg ICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgNwogICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAg ICAgIDUKICAgICAgICBiRW5kcG9pbnRBZGRyZXNzICAgICAweDgxICBFUCAxIElOCiAgICAgICAg Ym1BdHRyaWJ1dGVzICAgICAgICAgICAgMwogICAgICAgICAgVHJhbnNmZXIgVHlwZSAgICAgICAg ICAgIEludGVycnVwdAogICAgICAgICAgU3luY2ggVHlwZSAgICAgICAgICAgICAgIE5vbmUKICAg ICAgICAgIFVzYWdlIFR5cGUgICAgICAgICAgICAgICBEYXRhCiAgICAgICAgd01heFBhY2tldFNp emUgICAgIDB4MDAwNCAgMXggNCBieXRlcwogICAgICAgIGJJbnRlcnZhbCAgICAgICAgICAgICAg MTAKRGV2aWNlIFN0YXR1czogICAgIDB4MDAwMAogIChCdXMgUG93ZXJlZCkKCkJ1cyAvZGV2L3Vz YiBEZXZpY2UgL2Rldi91Z2VuMC4xOiBJRCAwMDAwOjAwMDAgIApEZXZpY2UgRGVzY3JpcHRvcjoK ICBiTGVuZ3RoICAgICAgICAgICAgICAgIDE4CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgMQog IGJjZFVTQiAgICAgICAgICAgICAgIDEuMDAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1 YgogIGJEZXZpY2VTdWJDbGFzcyAgICAgICAgIDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAg ICAgICAgMCBGdWxsIHNwZWVkIChvciByb290KSBodWIKICBiTWF4UGFja2V0U2l6ZTAgICAgICAg IDY0CiAgaWRWZW5kb3IgICAgICAgICAgIDB4MDAwMCAKICBpZFByb2R1Y3QgICAgICAgICAgMHgw MDAwIAogIGJjZERldmljZSAgICAgICAgICAgIDEuMDAKICBpTWFudWZhY3R1cmVyICAgICAgICAg ICAxIEFUSQogIGlQcm9kdWN0ICAgICAgICAgICAgICAgIDIgT0hDSSByb290IEhVQgogIGlTZXJp YWwgICAgICAgICAgICAgICAgIDAgCiAgYk51bUNvbmZpZ3VyYXRpb25zICAgICAgMQogIENvbmZp Z3VyYXRpb24gRGVzY3JpcHRvcjoKICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDkKICAgIGJE ZXNjcmlwdG9yVHlwZSAgICAgICAgIDIKICAgIHdUb3RhbExlbmd0aCAgICAgICAgICAgMjUKICAg IGJOdW1JbnRlcmZhY2VzICAgICAgICAgIDEKICAgIGJDb25maWd1cmF0aW9uVmFsdWUgICAgIDEK ICAgIGlDb25maWd1cmF0aW9uICAgICAgICAgIDAgCiAgICBibUF0dHJpYnV0ZXMgICAgICAgICAw eDQwCiAgICAgIChNaXNzaW5nIG11c3QtYmUtc2V0IGJpdCEpCiAgICAgIFNlbGYgUG93ZXJlZAog ICAgTWF4UG93ZXIgICAgICAgICAgICAgICAgMG1BCiAgICBJbnRlcmZhY2UgRGVzY3JpcHRvcjoK ICAgICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgICBiRGVzY3JpcHRvclR5cGUgICAg ICAgICA0CiAgICAgIGJJbnRlcmZhY2VOdW1iZXIgICAgICAgIDAKICAgICAgYkFsdGVybmF0ZVNl dHRpbmcgICAgICAgMAogICAgICBiTnVtRW5kcG9pbnRzICAgICAgICAgICAxCiAgICAgIGJJbnRl cmZhY2VDbGFzcyAgICAgICAgIDkgSHViCiAgICAgIGJJbnRlcmZhY2VTdWJDbGFzcyAgICAgIDAg VW51c2VkCiAgICAgIGJJbnRlcmZhY2VQcm90b2NvbCAgICAgIDAgRnVsbCBzcGVlZCAob3Igcm9v dCkgaHViCiAgICAgIGlJbnRlcmZhY2UgICAgICAgICAgICAgIDAgCiAgICAgIEVuZHBvaW50IERl c2NyaXB0b3I6CiAgICAgICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgNwogICAgICAgIGJEZXNj cmlwdG9yVHlwZSAgICAgICAgIDUKICAgICAgICBiRW5kcG9pbnRBZGRyZXNzICAgICAweDgxICBF UCAxIElOCiAgICAgICAgYm1BdHRyaWJ1dGVzICAgICAgICAgICAgMwogICAgICAgICAgVHJhbnNm ZXIgVHlwZSAgICAgICAgICAgIEludGVycnVwdAogICAgICAgICAgU3luY2ggVHlwZSAgICAgICAg ICAgICAgIE5vbmUKICAgICAgICAgIFVzYWdlIFR5cGUgICAgICAgICAgICAgICBEYXRhCiAgICAg ICAgd01heFBhY2tldFNpemUgICAgIDB4MDAyMCAgMXggMzIgYnl0ZXMKICAgICAgICBiSW50ZXJ2 YWwgICAgICAgICAgICAgMjU1Ckh1YiBEZXNjcmlwdG9yOgogIGJMZW5ndGggICAgICAgICAgICAg ICA5CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgNDEKICBuTmJyUG9ydHMgICAgICAgICAgICAgNQog IHdIdWJDaGFyYWN0ZXJpc3RpYyAweDAwMDAKICAgIEdhbmdlZCBwb3dlciBzd2l0Y2hpbmcKICAg IEdhbmdlZCBvdmVyY3VycmVudCBwcm90ZWN0aW9uCiAgYlB3ck9uMlB3ckdvb2QgICAgICAgIDIg KiAyIG1pbGxpIHNlY29uZHMKICBiSHViQ29udHJDdXJyZW50ICAgICAgMCBtaWxsaSBBbXBlcmUK ICBEZXZpY2VSZW1vdmFibGUgICAgMHgwMAogIFBvcnRQd3JDdHJsTWFzayAgICAweDAwCiBIdWIg UG9ydCBTdGF0dXM6CiAgIFBvcnQgMTogMDAwMC4wMTAwIHBvd2VyCiAgIFBvcnQgMjogMDAwMC4w MTAwIHBvd2VyCiAgIFBvcnQgMzogMDAwMC4wMTAwIHBvd2VyCiAgIFBvcnQgNDogMDAwMC4wMTAw IHBvd2VyCiAgIFBvcnQgNTogMDAwMC4wMTAwIHBvd2VyCkRldmljZSBTdGF0dXM6ICAgICAweDAw MDEKICBTZWxmIFBvd2VyZWQKCkJ1cyAvZGV2L3VzYiBEZXZpY2UgL2Rldi91Z2VuNC4xOiBJRCAw MDAwOjAwMDAgIApEZXZpY2UgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAgICAgICAgICAgIDE4 CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgMQogIGJjZFVTQiAgICAgICAgICAgICAgIDEuMDAK ICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAgICAgICAg IDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAgICAgICAgMCBGdWxsIHNwZWVkIChvciByb290 KSBodWIKICBiTWF4UGFja2V0U2l6ZTAgICAgICAgIDY0CiAgaWRWZW5kb3IgICAgICAgICAgIDB4 MDAwMCAKICBpZFByb2R1Y3QgICAgICAgICAgMHgwMDAwIAogIGJjZERldmljZSAgICAgICAgICAg IDEuMDAKICBpTWFudWZhY3R1cmVyICAgICAgICAgICAxIEFUSQogIGlQcm9kdWN0ICAgICAgICAg ICAgICAgIDIgT0hDSSByb290IEhVQgogIGlTZXJpYWwgICAgICAgICAgICAgICAgIDAgCiAgYk51 bUNvbmZpZ3VyYXRpb25zICAgICAgMQogIENvbmZpZ3VyYXRpb24gRGVzY3JpcHRvcjoKICAgIGJM ZW5ndGggICAgICAgICAgICAgICAgIDkKICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDIKICAg IHdUb3RhbExlbmd0aCAgICAgICAgICAgMjUKICAgIGJOdW1JbnRlcmZhY2VzICAgICAgICAgIDEK ICAgIGJDb25maWd1cmF0aW9uVmFsdWUgICAgIDEKICAgIGlDb25maWd1cmF0aW9uICAgICAgICAg IDAgCiAgICBibUF0dHJpYnV0ZXMgICAgICAgICAweDQwCiAgICAgIChNaXNzaW5nIG11c3QtYmUt c2V0IGJpdCEpCiAgICAgIFNlbGYgUG93ZXJlZAogICAgTWF4UG93ZXIgICAgICAgICAgICAgICAg MG1BCiAgICBJbnRlcmZhY2UgRGVzY3JpcHRvcjoKICAgICAgYkxlbmd0aCAgICAgICAgICAgICAg ICAgOQogICAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgICA0CiAgICAgIGJJbnRlcmZhY2VOdW1i ZXIgICAgICAgIDAKICAgICAgYkFsdGVybmF0ZVNldHRpbmcgICAgICAgMAogICAgICBiTnVtRW5k cG9pbnRzICAgICAgICAgICAxCiAgICAgIGJJbnRlcmZhY2VDbGFzcyAgICAgICAgIDkgSHViCiAg ICAgIGJJbnRlcmZhY2VTdWJDbGFzcyAgICAgIDAgVW51c2VkCiAgICAgIGJJbnRlcmZhY2VQcm90 b2NvbCAgICAgIDAgRnVsbCBzcGVlZCAob3Igcm9vdCkgaHViCiAgICAgIGlJbnRlcmZhY2UgICAg ICAgICAgICAgIDAgCiAgICAgIEVuZHBvaW50IERlc2NyaXB0b3I6CiAgICAgICAgYkxlbmd0aCAg ICAgICAgICAgICAgICAgNwogICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDUKICAgICAg ICBiRW5kcG9pbnRBZGRyZXNzICAgICAweDgxICBFUCAxIElOCiAgICAgICAgYm1BdHRyaWJ1dGVz ICAgICAgICAgICAgMwogICAgICAgICAgVHJhbnNmZXIgVHlwZSAgICAgICAgICAgIEludGVycnVw dAogICAgICAgICAgU3luY2ggVHlwZSAgICAgICAgICAgICAgIE5vbmUKICAgICAgICAgIFVzYWdl IFR5cGUgICAgICAgICAgICAgICBEYXRhCiAgICAgICAgd01heFBhY2tldFNpemUgICAgIDB4MDAy MCAgMXggMzIgYnl0ZXMKICAgICAgICBiSW50ZXJ2YWwgICAgICAgICAgICAgMjU1Ckh1YiBEZXNj cmlwdG9yOgogIGJMZW5ndGggICAgICAgICAgICAgICA5CiAgYkRlc2NyaXB0b3JUeXBlICAgICAg NDEKICBuTmJyUG9ydHMgICAgICAgICAgICAgMgogIHdIdWJDaGFyYWN0ZXJpc3RpYyAweDAwMDAK ICAgIEdhbmdlZCBwb3dlciBzd2l0Y2hpbmcKICAgIEdhbmdlZCBvdmVyY3VycmVudCBwcm90ZWN0 aW9uCiAgYlB3ck9uMlB3ckdvb2QgICAgICAgIDIgKiAyIG1pbGxpIHNlY29uZHMKICBiSHViQ29u dHJDdXJyZW50ICAgICAgMCBtaWxsaSBBbXBlcmUKICBEZXZpY2VSZW1vdmFibGUgICAgMHgwMAog IFBvcnRQd3JDdHJsTWFzayAgICAweDAwCiBIdWIgUG9ydCBTdGF0dXM6CiAgIFBvcnQgMTogMDAw MC4wMTAwIHBvd2VyCiAgIFBvcnQgMjogMDAwMC4wMTAwIHBvd2VyCkRldmljZSBTdGF0dXM6ICAg ICAweDAwMDEKICBTZWxmIFBvd2VyZWQKCkJ1cyAvZGV2L3VzYiBEZXZpY2UgL2Rldi91Z2VuNy4x OiBJRCAwMDAwOjAwMDAgIApEZXZpY2UgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAgICAgICAg ICAgIDE4CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgMQogIGJjZFVTQiAgICAgICAgICAgICAg IDIuMDAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAg ICAgICAgIDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAgICAgICAgMSBTaW5nbGUgVFQKICBi TWF4UGFja2V0U2l6ZTAgICAgICAgIDY0CiAgaWRWZW5kb3IgICAgICAgICAgIDB4MDAwMCAKICBp ZFByb2R1Y3QgICAgICAgICAgMHgwMDAwIAogIGJjZERldmljZSAgICAgICAgICAgIDEuMDAKICBp TWFudWZhY3R1cmVyICAgICAgICAgICAxIEFUSQogIGlQcm9kdWN0ICAgICAgICAgICAgICAgIDIg RUhDSSByb290IEhVQgogIGlTZXJpYWwgICAgICAgICAgICAgICAgIDAgCiAgYk51bUNvbmZpZ3Vy YXRpb25zICAgICAgMQogIENvbmZpZ3VyYXRpb24gRGVzY3JpcHRvcjoKICAgIGJMZW5ndGggICAg ICAgICAgICAgICAgIDkKICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDIKICAgIHdUb3RhbExl bmd0aCAgICAgICAgICAgMjUKICAgIGJOdW1JbnRlcmZhY2VzICAgICAgICAgIDEKICAgIGJDb25m aWd1cmF0aW9uVmFsdWUgICAgIDEKICAgIGlDb25maWd1cmF0aW9uICAgICAgICAgIDAgCiAgICBi bUF0dHJpYnV0ZXMgICAgICAgICAweDQwCiAgICAgIChNaXNzaW5nIG11c3QtYmUtc2V0IGJpdCEp CiAgICAgIFNlbGYgUG93ZXJlZAogICAgTWF4UG93ZXIgICAgICAgICAgICAgICAgMG1BCiAgICBJ bnRlcmZhY2UgRGVzY3JpcHRvcjoKICAgICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAg ICBiRGVzY3JpcHRvclR5cGUgICAgICAgICA0CiAgICAgIGJJbnRlcmZhY2VOdW1iZXIgICAgICAg IDAKICAgICAgYkFsdGVybmF0ZVNldHRpbmcgICAgICAgMAogICAgICBiTnVtRW5kcG9pbnRzICAg ICAgICAgICAxCiAgICAgIGJJbnRlcmZhY2VDbGFzcyAgICAgICAgIDkgSHViCiAgICAgIGJJbnRl cmZhY2VTdWJDbGFzcyAgICAgIDAgVW51c2VkCiAgICAgIGJJbnRlcmZhY2VQcm90b2NvbCAgICAg IDAgRnVsbCBzcGVlZCAob3Igcm9vdCkgaHViCiAgICAgIGlJbnRlcmZhY2UgICAgICAgICAgICAg IDAgCiAgICAgIEVuZHBvaW50IERlc2NyaXB0b3I6CiAgICAgICAgYkxlbmd0aCAgICAgICAgICAg ICAgICAgNwogICAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDUKICAgICAgICBiRW5kcG9p bnRBZGRyZXNzICAgICAweDgxICBFUCAxIElOCiAgICAgICAgYm1BdHRyaWJ1dGVzICAgICAgICAg ICAgMwogICAgICAgICAgVHJhbnNmZXIgVHlwZSAgICAgICAgICAgIEludGVycnVwdAogICAgICAg ICAgU3luY2ggVHlwZSAgICAgICAgICAgICAgIE5vbmUKICAgICAgICAgIFVzYWdlIFR5cGUgICAg ICAgICAgICAgICBEYXRhCiAgICAgICAgd01heFBhY2tldFNpemUgICAgIDB4MDAwOCAgMXggOCBi eXRlcwogICAgICAgIGJJbnRlcnZhbCAgICAgICAgICAgICAyNTUKSHViIERlc2NyaXB0b3I6CiAg Ykxlbmd0aCAgICAgICAgICAgICAgIDkKICBiRGVzY3JpcHRvclR5cGUgICAgICA0MQogIG5OYnJQ b3J0cyAgICAgICAgICAgICA0CiAgd0h1YkNoYXJhY3RlcmlzdGljIDB4MDAwMgogICAgTm8gcG93 ZXIgc3dpdGNoaW5nICh1c2IgMS4wKQogICAgR2FuZ2VkIG92ZXJjdXJyZW50IHByb3RlY3Rpb24K ICAgIFRUIHRoaW5rIHRpbWUgOCBGUyBiaXRzCiAgYlB3ck9uMlB3ckdvb2QgICAgICAyMDAgKiAy IG1pbGxpIHNlY29uZHMKICBiSHViQ29udHJDdXJyZW50ICAgICAgMCBtaWxsaSBBbXBlcmUKICBE ZXZpY2VSZW1vdmFibGUgICAgMHgwMAogIFBvcnRQd3JDdHJsTWFzayAgICAweDAwCiBIdWIgUG9y dCBTdGF0dXM6CiAgIFBvcnQgMTogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgogICBQb3J0IDI6 IDAwMDAuMDUwMCBoaWdoc3BlZWQgcG93ZXIKICAgUG9ydCAzOiAwMDAwLjA1MDAgaGlnaHNwZWVk IHBvd2VyCiAgIFBvcnQgNDogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgpEZXZpY2UgUXVhbGlm aWVyIChmb3Igb3RoZXIgZGV2aWNlIHNwZWVkKToKICBiTGVuZ3RoICAgICAgICAgICAgICAgIDEw CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgNgogIGJjZFVTQiAgICAgICAgICAgICAgIDIuMDAK ICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAgICAgICAg IDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAgICAgICAgMCBGdWxsIHNwZWVkIChvciByb290 KSBodWIKICBiTWF4UGFja2V0U2l6ZTAgICAgICAgICAwCiAgYk51bUNvbmZpZ3VyYXRpb25zICAg ICAgMApEZXZpY2UgU3RhdHVzOiAgICAgMHgwMDAxCiAgU2VsZiBQb3dlcmVkCgpCdXMgL2Rldi91 c2IgRGV2aWNlIC9kZXYvdWdlbjMuMTogSUQgMDAwMDowMDAwICAKRGV2aWNlIERlc2NyaXB0b3I6 CiAgYkxlbmd0aCAgICAgICAgICAgICAgICAxOAogIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDEK ICBiY2RVU0IgICAgICAgICAgICAgICAyLjAwCiAgYkRldmljZUNsYXNzICAgICAgICAgICAgOSBI dWIKICBiRGV2aWNlU3ViQ2xhc3MgICAgICAgICAwIFVudXNlZAogIGJEZXZpY2VQcm90b2NvbCAg ICAgICAgIDEgU2luZ2xlIFRUCiAgYk1heFBhY2tldFNpemUwICAgICAgICA2NAogIGlkVmVuZG9y ICAgICAgICAgICAweDAwMDAgCiAgaWRQcm9kdWN0ICAgICAgICAgIDB4MDAwMCAKICBiY2REZXZp Y2UgICAgICAgICAgICAxLjAwCiAgaU1hbnVmYWN0dXJlciAgICAgICAgICAgMSBBVEkKICBpUHJv ZHVjdCAgICAgICAgICAgICAgICAyIEVIQ0kgcm9vdCBIVUIKICBpU2VyaWFsICAgICAgICAgICAg ICAgICAwIAogIGJOdW1Db25maWd1cmF0aW9ucyAgICAgIDEKICBDb25maWd1cmF0aW9uIERlc2Ny aXB0b3I6CiAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA5CiAgICBiRGVzY3JpcHRvclR5cGUg ICAgICAgICAyCiAgICB3VG90YWxMZW5ndGggICAgICAgICAgIDI1CiAgICBiTnVtSW50ZXJmYWNl cyAgICAgICAgICAxCiAgICBiQ29uZmlndXJhdGlvblZhbHVlICAgICAxCiAgICBpQ29uZmlndXJh dGlvbiAgICAgICAgICAwIAogICAgYm1BdHRyaWJ1dGVzICAgICAgICAgMHg0MAogICAgICAoTWlz c2luZyBtdXN0LWJlLXNldCBiaXQhKQogICAgICBTZWxmIFBvd2VyZWQKICAgIE1heFBvd2VyICAg ICAgICAgICAgICAgIDBtQQogICAgSW50ZXJmYWNlIERlc2NyaXB0b3I6CiAgICAgIGJMZW5ndGgg ICAgICAgICAgICAgICAgIDkKICAgICAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgNAogICAgICBi SW50ZXJmYWNlTnVtYmVyICAgICAgICAwCiAgICAgIGJBbHRlcm5hdGVTZXR0aW5nICAgICAgIDAK ICAgICAgYk51bUVuZHBvaW50cyAgICAgICAgICAgMQogICAgICBiSW50ZXJmYWNlQ2xhc3MgICAg ICAgICA5IEh1YgogICAgICBiSW50ZXJmYWNlU3ViQ2xhc3MgICAgICAwIFVudXNlZAogICAgICBi SW50ZXJmYWNlUHJvdG9jb2wgICAgICAwIEZ1bGwgc3BlZWQgKG9yIHJvb3QpIGh1YgogICAgICBp SW50ZXJmYWNlICAgICAgICAgICAgICAwIAogICAgICBFbmRwb2ludCBEZXNjcmlwdG9yOgogICAg ICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDcKICAgICAgICBiRGVzY3JpcHRvclR5cGUgICAg ICAgICA1CiAgICAgICAgYkVuZHBvaW50QWRkcmVzcyAgICAgMHg4MSAgRVAgMSBJTgogICAgICAg IGJtQXR0cmlidXRlcyAgICAgICAgICAgIDMKICAgICAgICAgIFRyYW5zZmVyIFR5cGUgICAgICAg ICAgICBJbnRlcnJ1cHQKICAgICAgICAgIFN5bmNoIFR5cGUgICAgICAgICAgICAgICBOb25lCiAg ICAgICAgICBVc2FnZSBUeXBlICAgICAgICAgICAgICAgRGF0YQogICAgICAgIHdNYXhQYWNrZXRT aXplICAgICAweDAwMDggIDF4IDggYnl0ZXMKICAgICAgICBiSW50ZXJ2YWwgICAgICAgICAgICAg MjU1Ckh1YiBEZXNjcmlwdG9yOgogIGJMZW5ndGggICAgICAgICAgICAgICA5CiAgYkRlc2NyaXB0 b3JUeXBlICAgICAgNDEKICBuTmJyUG9ydHMgICAgICAgICAgICAgNQogIHdIdWJDaGFyYWN0ZXJp c3RpYyAweDAwMDIKICAgIE5vIHBvd2VyIHN3aXRjaGluZyAodXNiIDEuMCkKICAgIEdhbmdlZCBv dmVyY3VycmVudCBwcm90ZWN0aW9uCiAgICBUVCB0aGluayB0aW1lIDggRlMgYml0cwogIGJQd3JP bjJQd3JHb29kICAgICAgMjAwICogMiBtaWxsaSBzZWNvbmRzCiAgYkh1YkNvbnRyQ3VycmVudCAg ICAgIDAgbWlsbGkgQW1wZXJlCiAgRGV2aWNlUmVtb3ZhYmxlICAgIDB4MDAKICBQb3J0UHdyQ3Ry bE1hc2sgICAgMHgwMAogSHViIFBvcnQgU3RhdHVzOgogICBQb3J0IDE6IDAwMDAuMDUwMCBoaWdo c3BlZWQgcG93ZXIKICAgUG9ydCAyOiAwMDAwLjA1MDAgaGlnaHNwZWVkIHBvd2VyCiAgIFBvcnQg MzogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgogICBQb3J0IDQ6IDAwMDAuMDUwMCBoaWdoc3Bl ZWQgcG93ZXIKICAgUG9ydCA1OiAwMDAwLjA1MDAgaGlnaHNwZWVkIHBvd2VyCkRldmljZSBRdWFs aWZpZXIgKGZvciBvdGhlciBkZXZpY2Ugc3BlZWQpOgogIGJMZW5ndGggICAgICAgICAgICAgICAg MTAKICBiRGVzY3JpcHRvclR5cGUgICAgICAgICA2CiAgYmNkVVNCICAgICAgICAgICAgICAgMi4w MAogIGJEZXZpY2VDbGFzcyAgICAgICAgICAgIDkgSHViCiAgYkRldmljZVN1YkNsYXNzICAgICAg ICAgMCBVbnVzZWQKICBiRGV2aWNlUHJvdG9jb2wgICAgICAgICAwIEZ1bGwgc3BlZWQgKG9yIHJv b3QpIGh1YgogIGJNYXhQYWNrZXRTaXplMCAgICAgICAgIDAKICBiTnVtQ29uZmlndXJhdGlvbnMg ICAgICAwCkRldmljZSBTdGF0dXM6ICAgICAweDAwMDEKICBTZWxmIFBvd2VyZWQKCkJ1cyAvZGV2 L3VzYiBEZXZpY2UgL2Rldi91Z2VuNS4xOiBJRCAwMDAwOjAwMDAgIApEZXZpY2UgRGVzY3JpcHRv cjoKICBiTGVuZ3RoICAgICAgICAgICAgICAgIDE4CiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAg MQogIGJjZFVTQiAgICAgICAgICAgICAgIDMuMDAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5 IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAgICAgICAgIDAgVW51c2VkCiAgYkRldmljZVByb3RvY29s ICAgICAgICAgMyAKICBiTWF4UGFja2V0U2l6ZTAgICAgICAgICA5CiAgaWRWZW5kb3IgICAgICAg ICAgIDB4MDAwMCAKICBpZFByb2R1Y3QgICAgICAgICAgMHgwMDAwIAogIGJjZERldmljZSAgICAg ICAgICAgIDEuMDAKICBpTWFudWZhY3R1cmVyICAgICAgICAgICAxICgweDFiNmYpCiAgaVByb2R1 Y3QgICAgICAgICAgICAgICAgMiBYSENJIHJvb3QgSFVCCiAgaVNlcmlhbCAgICAgICAgICAgICAg ICAgMCAKICBiTnVtQ29uZmlndXJhdGlvbnMgICAgICAxCiAgQ29uZmlndXJhdGlvbiBEZXNjcmlw dG9yOgogICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgYkRlc2NyaXB0b3JUeXBlICAg ICAgICAgMgogICAgd1RvdGFsTGVuZ3RoICAgICAgICAgICAzMQogICAgYk51bUludGVyZmFjZXMg ICAgICAgICAgMQogICAgYkNvbmZpZ3VyYXRpb25WYWx1ZSAgICAgMQogICAgaUNvbmZpZ3VyYXRp b24gICAgICAgICAgMCAKICAgIGJtQXR0cmlidXRlcyAgICAgICAgIDB4NDAKICAgICAgKE1pc3Np bmcgbXVzdC1iZS1zZXQgYml0ISkKICAgICAgU2VsZiBQb3dlcmVkCiAgICBNYXhQb3dlciAgICAg ICAgICAgICAgICAwbUEKICAgIEludGVyZmFjZSBEZXNjcmlwdG9yOgogICAgICBiTGVuZ3RoICAg ICAgICAgICAgICAgICA5CiAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDQKICAgICAgYklu dGVyZmFjZU51bWJlciAgICAgICAgMAogICAgICBiQWx0ZXJuYXRlU2V0dGluZyAgICAgICAwCiAg ICAgIGJOdW1FbmRwb2ludHMgICAgICAgICAgIDEKICAgICAgYkludGVyZmFjZUNsYXNzICAgICAg ICAgOSBIdWIKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAgMCBVbnVzZWQKICAgICAgYklu dGVyZmFjZVByb3RvY29sICAgICAgMCBGdWxsIHNwZWVkIChvciByb290KSBodWIKICAgICAgaUlu dGVyZmFjZSAgICAgICAgICAgICAgMCAKICAgICAgRW5kcG9pbnQgRGVzY3JpcHRvcjoKICAgICAg ICBiTGVuZ3RoICAgICAgICAgICAgICAgICA3CiAgICAgICAgYkRlc2NyaXB0b3JUeXBlICAgICAg ICAgNQogICAgICAgIGJFbmRwb2ludEFkZHJlc3MgICAgIDB4ODEgIEVQIDEgSU4KICAgICAgICBi bUF0dHJpYnV0ZXMgICAgICAgICAgICAzCiAgICAgICAgICBUcmFuc2ZlciBUeXBlICAgICAgICAg ICAgSW50ZXJydXB0CiAgICAgICAgICBTeW5jaCBUeXBlICAgICAgICAgICAgICAgTm9uZQogICAg ICAgICAgVXNhZ2UgVHlwZSAgICAgICAgICAgICAgIERhdGEKICAgICAgICB3TWF4UGFja2V0U2l6 ZSAgICAgMHgwMDAyICAxeCAyIGJ5dGVzCiAgICAgICAgYkludGVydmFsICAgICAgICAgICAgIDI1 NQogICAgICAgIGJNYXhCdXJzdCAgICAgICAgICAgICAgIDEKSHViIERlc2NyaXB0b3I6CiAgYkxl bmd0aCAgICAgICAgICAgICAgNDIKICBiRGVzY3JpcHRvclR5cGUgICAgICA0MgogIG5OYnJQb3J0 cyAgICAgICAgICAgICA0CiAgd0h1YkNoYXJhY3RlcmlzdGljIDB4MDAwOQogICAgUGVyLXBvcnQg cG93ZXIgc3dpdGNoaW5nCiAgICBQZXItcG9ydCBvdmVyY3VycmVudCBwcm90ZWN0aW9uCiAgYlB3 ck9uMlB3ckdvb2QgICAgICAgMTAgKiAyIG1pbGxpIHNlY29uZHMKICBiSHViQ29udHJDdXJyZW50 ICAgICAgMCBtaWxsaSBBbXBlcmUKICBiSHViRGVjTGF0ICAgICAgICAgIDAuMCBtaWNybyBzZWNv bmRzCiAgd0h1YkRlbGF5ICAgICAgICAgICAgIDAgbmFubyBzZWNvbmRzCiAgRGV2aWNlUmVtb3Zh YmxlICAgIDB4MDAKIEh1YiBQb3J0IFN0YXR1czoKICAgUG9ydCAxOiAwMDAwLjA2YTAgVW5rbm93 biBTcGVlZCBwb3dlciBSeC5EZXRlY3QKICAgUG9ydCAyOiAwMDAwLjA2YTAgVW5rbm93biBTcGVl ZCBwb3dlciBSeC5EZXRlY3QKICAgUG9ydCAzOiAwMDAwLjA2YTAgVW5rbm93biBTcGVlZCBwb3dl ciBSeC5EZXRlY3QKICAgUG9ydCA0OiAwMDAwLjA2YTAgVW5rbm93biBTcGVlZCBwb3dlciBSeC5E ZXRlY3QKRGV2aWNlIFN0YXR1czogICAgIDB4MDAwMQogIFNlbGYgUG93ZXJlZApCaW5hcnkgT2Jq ZWN0IFN0b3JlIERlc2NyaXB0b3I6CiAgYkxlbmd0aCAgICAgICAgICAgICAgICAgNQogIGJEZXNj cmlwdG9yVHlwZSAgICAgICAgMTUKICB3VG90YWxMZW5ndGggICAgICAgICAgIDI3CiAgYk51bURl dmljZUNhcHMgICAgICAgICAgMwogIFVTQiAyLjAgRXh0ZW5zaW9uIERldmljZSBDYXBhYmlsaXR5 OgogICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgNwogICAgYkRlc2NyaXB0b3JUeXBlICAgICAg ICAgMQogICAgYkRldkNhcGFiaWxpdHlUeXBlICAgICAgMgogICAgYm1BdHRyaWJ1dGVzICAgMHgw MDAwMDAwMgogICAgICBMaW5rIFBvd2VyIE1hbmFnZW1lbnQgKExQTSkgU3VwcG9ydGVkCiAgU3Vw ZXJTcGVlZCBVU0IgRGV2aWNlIENhcGFiaWxpdHk6CiAgICBiTGVuZ3RoICAgICAgICAgICAgICAg IDEwCiAgICBiRGVzY3JpcHRvclR5cGUgICAgICAgIDE2CiAgICBiRGV2Q2FwYWJpbGl0eVR5cGUg ICAgICAzCiAgICBibUF0dHJpYnV0ZXMgICAgICAgICAweDAwCiAgICAgIExhdGVuY3kgVG9sZXJh bmNlIE1lc3NhZ2VzIChMVE0pIFN1cHBvcnRlZAogICAgd1NwZWVkc1N1cHBvcnRlZCAgIDB4MDAw YwogICAgICBEZXZpY2UgY2FuIG9wZXJhdGUgYXQgSGlnaCBTcGVlZCAoNDgwTWJwcykKICAgICAg RGV2aWNlIGNhbiBvcGVyYXRlIGF0IFN1cGVyU3BlZWQgKDVHYnBzKQogICAgYkZ1bmN0aW9uYWxp dHlTdXBwb3J0ICAgMAogICAgICBMb3dlc3QgZnVsbHktZnVuY3Rpb25hbCBkZXZpY2Ugc3BlZWQg aXMgTG93IFNwZWVkICgxTWJwcykKICAgIGJVMURldkV4aXRMYXQgICAgICAgICAgIDggbWljcm8g c2Vjb25kcwogICAgYlUyRGV2RXhpdExhdCAgICAgICA2NTI4MCBtaWNybyBzZWNvbmRzCiAgQmFk IENvbnRhaW5lciBJRCBEZXZpY2UgQ2FwYWJpbGl0eSBkZXNjcmlwdG9yLgoKQnVzIC9kZXYvdXNi IERldmljZSAvZGV2L3VnZW42LjE6IElEIDAwMDA6MDAwMCAgCkRldmljZSBEZXNjcmlwdG9yOgog IGJMZW5ndGggICAgICAgICAgICAgICAgMTgKICBiRGVzY3JpcHRvclR5cGUgICAgICAgICAxCiAg YmNkVVNCICAgICAgICAgICAgICAgMS4wMAogIGJEZXZpY2VDbGFzcyAgICAgICAgICAgIDkgSHVi CiAgYkRldmljZVN1YkNsYXNzICAgICAgICAgMCBVbnVzZWQKICBiRGV2aWNlUHJvdG9jb2wgICAg ICAgICAwIEZ1bGwgc3BlZWQgKG9yIHJvb3QpIGh1YgogIGJNYXhQYWNrZXRTaXplMCAgICAgICAg NjQKICBpZFZlbmRvciAgICAgICAgICAgMHgwMDAwIAogIGlkUHJvZHVjdCAgICAgICAgICAweDAw MDAgCiAgYmNkRGV2aWNlICAgICAgICAgICAgMS4wMAogIGlNYW51ZmFjdHVyZXIgICAgICAgICAg IDEgQVRJCiAgaVByb2R1Y3QgICAgICAgICAgICAgICAgMiBPSENJIHJvb3QgSFVCCiAgaVNlcmlh bCAgICAgICAgICAgICAgICAgMCAKICBiTnVtQ29uZmlndXJhdGlvbnMgICAgICAxCiAgQ29uZmln dXJhdGlvbiBEZXNjcmlwdG9yOgogICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgYkRl c2NyaXB0b3JUeXBlICAgICAgICAgMgogICAgd1RvdGFsTGVuZ3RoICAgICAgICAgICAyNQogICAg Yk51bUludGVyZmFjZXMgICAgICAgICAgMQogICAgYkNvbmZpZ3VyYXRpb25WYWx1ZSAgICAgMQog ICAgaUNvbmZpZ3VyYXRpb24gICAgICAgICAgMCAKICAgIGJtQXR0cmlidXRlcyAgICAgICAgIDB4 NDAKICAgICAgKE1pc3NpbmcgbXVzdC1iZS1zZXQgYml0ISkKICAgICAgU2VsZiBQb3dlcmVkCiAg ICBNYXhQb3dlciAgICAgICAgICAgICAgICAwbUEKICAgIEludGVyZmFjZSBEZXNjcmlwdG9yOgog ICAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA5CiAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAg ICAgIDQKICAgICAgYkludGVyZmFjZU51bWJlciAgICAgICAgMAogICAgICBiQWx0ZXJuYXRlU2V0 dGluZyAgICAgICAwCiAgICAgIGJOdW1FbmRwb2ludHMgICAgICAgICAgIDEKICAgICAgYkludGVy ZmFjZUNsYXNzICAgICAgICAgOSBIdWIKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAgMCBV bnVzZWQKICAgICAgYkludGVyZmFjZVByb3RvY29sICAgICAgMCBGdWxsIHNwZWVkIChvciByb290 KSBodWIKICAgICAgaUludGVyZmFjZSAgICAgICAgICAgICAgMCAKICAgICAgRW5kcG9pbnQgRGVz Y3JpcHRvcjoKICAgICAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA3CiAgICAgICAgYkRlc2Ny aXB0b3JUeXBlICAgICAgICAgNQogICAgICAgIGJFbmRwb2ludEFkZHJlc3MgICAgIDB4ODEgIEVQ IDEgSU4KICAgICAgICBibUF0dHJpYnV0ZXMgICAgICAgICAgICAzCiAgICAgICAgICBUcmFuc2Zl ciBUeXBlICAgICAgICAgICAgSW50ZXJydXB0CiAgICAgICAgICBTeW5jaCBUeXBlICAgICAgICAg ICAgICAgTm9uZQogICAgICAgICAgVXNhZ2UgVHlwZSAgICAgICAgICAgICAgIERhdGEKICAgICAg ICB3TWF4UGFja2V0U2l6ZSAgICAgMHgwMDIwICAxeCAzMiBieXRlcwogICAgICAgIGJJbnRlcnZh bCAgICAgICAgICAgICAyNTUKSHViIERlc2NyaXB0b3I6CiAgYkxlbmd0aCAgICAgICAgICAgICAg IDkKICBiRGVzY3JpcHRvclR5cGUgICAgICA0MQogIG5OYnJQb3J0cyAgICAgICAgICAgICA0CiAg d0h1YkNoYXJhY3RlcmlzdGljIDB4MDAwMAogICAgR2FuZ2VkIHBvd2VyIHN3aXRjaGluZwogICAg R2FuZ2VkIG92ZXJjdXJyZW50IHByb3RlY3Rpb24KICBiUHdyT24yUHdyR29vZCAgICAgICAgMiAq IDIgbWlsbGkgc2Vjb25kcwogIGJIdWJDb250ckN1cnJlbnQgICAgICAwIG1pbGxpIEFtcGVyZQog IERldmljZVJlbW92YWJsZSAgICAweDAwCiAgUG9ydFB3ckN0cmxNYXNrICAgIDB4MDAKIEh1YiBQ b3J0IFN0YXR1czoKICAgUG9ydCAxOiAwMDAwLjAxMDAgcG93ZXIKICAgUG9ydCAyOiAwMDAwLjAx MDAgcG93ZXIKICAgUG9ydCAzOiAwMDAwLjAxMDAgcG93ZXIKICAgUG9ydCA0OiAwMDAwLjAxMDAg cG93ZXIKRGV2aWNlIFN0YXR1czogICAgIDB4MDAwMQogIFNlbGYgUG93ZXJlZAoKQnVzIC9kZXYv dXNiIERldmljZSAvZGV2L3VnZW4xLjE6IElEIDAwMDA6MDAwMCAgCkRldmljZSBEZXNjcmlwdG9y OgogIGJMZW5ndGggICAgICAgICAgICAgICAgMTgKICBiRGVzY3JpcHRvclR5cGUgICAgICAgICAx CiAgYmNkVVNCICAgICAgICAgICAgICAgMi4wMAogIGJEZXZpY2VDbGFzcyAgICAgICAgICAgIDkg SHViCiAgYkRldmljZVN1YkNsYXNzICAgICAgICAgMCBVbnVzZWQKICBiRGV2aWNlUHJvdG9jb2wg ICAgICAgICAxIFNpbmdsZSBUVAogIGJNYXhQYWNrZXRTaXplMCAgICAgICAgNjQKICBpZFZlbmRv ciAgICAgICAgICAgMHgwMDAwIAogIGlkUHJvZHVjdCAgICAgICAgICAweDAwMDAgCiAgYmNkRGV2 aWNlICAgICAgICAgICAgMS4wMAogIGlNYW51ZmFjdHVyZXIgICAgICAgICAgIDEgQVRJCiAgaVBy b2R1Y3QgICAgICAgICAgICAgICAgMiBFSENJIHJvb3QgSFVCCiAgaVNlcmlhbCAgICAgICAgICAg ICAgICAgMCAKICBiTnVtQ29uZmlndXJhdGlvbnMgICAgICAxCiAgQ29uZmlndXJhdGlvbiBEZXNj cmlwdG9yOgogICAgYkxlbmd0aCAgICAgICAgICAgICAgICAgOQogICAgYkRlc2NyaXB0b3JUeXBl ICAgICAgICAgMgogICAgd1RvdGFsTGVuZ3RoICAgICAgICAgICAyNQogICAgYk51bUludGVyZmFj ZXMgICAgICAgICAgMQogICAgYkNvbmZpZ3VyYXRpb25WYWx1ZSAgICAgMQogICAgaUNvbmZpZ3Vy YXRpb24gICAgICAgICAgMCAKICAgIGJtQXR0cmlidXRlcyAgICAgICAgIDB4NDAKICAgICAgKE1p c3NpbmcgbXVzdC1iZS1zZXQgYml0ISkKICAgICAgU2VsZiBQb3dlcmVkCiAgICBNYXhQb3dlciAg ICAgICAgICAgICAgICAwbUEKICAgIEludGVyZmFjZSBEZXNjcmlwdG9yOgogICAgICBiTGVuZ3Ro ICAgICAgICAgICAgICAgICA5CiAgICAgIGJEZXNjcmlwdG9yVHlwZSAgICAgICAgIDQKICAgICAg YkludGVyZmFjZU51bWJlciAgICAgICAgMAogICAgICBiQWx0ZXJuYXRlU2V0dGluZyAgICAgICAw CiAgICAgIGJOdW1FbmRwb2ludHMgICAgICAgICAgIDEKICAgICAgYkludGVyZmFjZUNsYXNzICAg ICAgICAgOSBIdWIKICAgICAgYkludGVyZmFjZVN1YkNsYXNzICAgICAgMCBVbnVzZWQKICAgICAg YkludGVyZmFjZVByb3RvY29sICAgICAgMCBGdWxsIHNwZWVkIChvciByb290KSBodWIKICAgICAg aUludGVyZmFjZSAgICAgICAgICAgICAgMCAKICAgICAgRW5kcG9pbnQgRGVzY3JpcHRvcjoKICAg ICAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA3CiAgICAgICAgYkRlc2NyaXB0b3JUeXBlICAg ICAgICAgNQogICAgICAgIGJFbmRwb2ludEFkZHJlc3MgICAgIDB4ODEgIEVQIDEgSU4KICAgICAg ICBibUF0dHJpYnV0ZXMgICAgICAgICAgICAzCiAgICAgICAgICBUcmFuc2ZlciBUeXBlICAgICAg ICAgICAgSW50ZXJydXB0CiAgICAgICAgICBTeW5jaCBUeXBlICAgICAgICAgICAgICAgTm9uZQog ICAgICAgICAgVXNhZ2UgVHlwZSAgICAgICAgICAgICAgIERhdGEKICAgICAgICB3TWF4UGFja2V0 U2l6ZSAgICAgMHgwMDA4ICAxeCA4IGJ5dGVzCiAgICAgICAgYkludGVydmFsICAgICAgICAgICAg IDI1NQpIdWIgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAgICAgICAgICAgOQogIGJEZXNjcmlw dG9yVHlwZSAgICAgIDQxCiAgbk5iclBvcnRzICAgICAgICAgICAgIDUKICB3SHViQ2hhcmFjdGVy aXN0aWMgMHgwMDAyCiAgICBObyBwb3dlciBzd2l0Y2hpbmcgKHVzYiAxLjApCiAgICBHYW5nZWQg b3ZlcmN1cnJlbnQgcHJvdGVjdGlvbgogICAgVFQgdGhpbmsgdGltZSA4IEZTIGJpdHMKICBiUHdy T24yUHdyR29vZCAgICAgIDIwMCAqIDIgbWlsbGkgc2Vjb25kcwogIGJIdWJDb250ckN1cnJlbnQg ICAgICAwIG1pbGxpIEFtcGVyZQogIERldmljZVJlbW92YWJsZSAgICAweDAwCiAgUG9ydFB3ckN0 cmxNYXNrICAgIDB4MDAKIEh1YiBQb3J0IFN0YXR1czoKICAgUG9ydCAxOiAwMDAwLjA1MDAgaGln aHNwZWVkIHBvd2VyCiAgIFBvcnQgMjogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgogICBQb3J0 IDM6IDAwMDAuMDUwMCBoaWdoc3BlZWQgcG93ZXIKICAgUG9ydCA0OiAwMDAwLjA1MDAgaGlnaHNw ZWVkIHBvd2VyCiAgIFBvcnQgNTogMDAwMC4wNTAwIGhpZ2hzcGVlZCBwb3dlcgpEZXZpY2UgUXVh bGlmaWVyIChmb3Igb3RoZXIgZGV2aWNlIHNwZWVkKToKICBiTGVuZ3RoICAgICAgICAgICAgICAg IDEwCiAgYkRlc2NyaXB0b3JUeXBlICAgICAgICAgNgogIGJjZFVTQiAgICAgICAgICAgICAgIDIu MDAKICBiRGV2aWNlQ2xhc3MgICAgICAgICAgICA5IEh1YgogIGJEZXZpY2VTdWJDbGFzcyAgICAg ICAgIDAgVW51c2VkCiAgYkRldmljZVByb3RvY29sICAgICAgICAgMCBGdWxsIHNwZWVkIChvciBy b290KSBodWIKICBiTWF4UGFja2V0U2l6ZTAgICAgICAgICAwCiAgYk51bUNvbmZpZ3VyYXRpb25z ICAgICAgMApEZXZpY2UgU3RhdHVzOiAgICAgMHgwMDAxCiAgU2VsZiBQb3dlcmVkCgpCdXMgL2Rl di91c2IgRGV2aWNlIC9kZXYvdWdlbjIuMTogSUQgMDAwMDowMDAwICAKRGV2aWNlIERlc2NyaXB0 b3I6CiAgYkxlbmd0aCAgICAgICAgICAgICAgICAxOAogIGJEZXNjcmlwdG9yVHlwZSAgICAgICAg IDEKICBiY2RVU0IgICAgICAgICAgICAgICAxLjAwCiAgYkRldmljZUNsYXNzICAgICAgICAgICAg OSBIdWIKICBiRGV2aWNlU3ViQ2xhc3MgICAgICAgICAwIFVudXNlZAogIGJEZXZpY2VQcm90b2Nv bCAgICAgICAgIDAgRnVsbCBzcGVlZCAob3Igcm9vdCkgaHViCiAgYk1heFBhY2tldFNpemUwICAg ICAgICA2NAogIGlkVmVuZG9yICAgICAgICAgICAweDAwMDAgCiAgaWRQcm9kdWN0ICAgICAgICAg IDB4MDAwMCAKICBiY2REZXZpY2UgICAgICAgICAgICAxLjAwCiAgaU1hbnVmYWN0dXJlciAgICAg ICAgICAgMSBBVEkKICBpUHJvZHVjdCAgICAgICAgICAgICAgICAyIE9IQ0kgcm9vdCBIVUIKICBp U2VyaWFsICAgICAgICAgICAgICAgICAwIAogIGJOdW1Db25maWd1cmF0aW9ucyAgICAgIDEKICBD b25maWd1cmF0aW9uIERlc2NyaXB0b3I6CiAgICBiTGVuZ3RoICAgICAgICAgICAgICAgICA5CiAg ICBiRGVzY3JpcHRvclR5cGUgICAgICAgICAyCiAgICB3VG90YWxMZW5ndGggICAgICAgICAgIDI1 CiAgICBiTnVtSW50ZXJmYWNlcyAgICAgICAgICAxCiAgICBiQ29uZmlndXJhdGlvblZhbHVlICAg ICAxCiAgICBpQ29uZmlndXJhdGlvbiAgICAgICAgICAwIAogICAgYm1BdHRyaWJ1dGVzICAgICAg ICAgMHg0MAogICAgICAoTWlzc2luZyBtdXN0LWJlLXNldCBiaXQhKQogICAgICBTZWxmIFBvd2Vy ZWQKICAgIE1heFBvd2VyICAgICAgICAgICAgICAgIDBtQQogICAgSW50ZXJmYWNlIERlc2NyaXB0 b3I6CiAgICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDkKICAgICAgYkRlc2NyaXB0b3JUeXBl ICAgICAgICAgNAogICAgICBiSW50ZXJmYWNlTnVtYmVyICAgICAgICAwCiAgICAgIGJBbHRlcm5h dGVTZXR0aW5nICAgICAgIDAKICAgICAgYk51bUVuZHBvaW50cyAgICAgICAgICAgMQogICAgICBi SW50ZXJmYWNlQ2xhc3MgICAgICAgICA5IEh1YgogICAgICBiSW50ZXJmYWNlU3ViQ2xhc3MgICAg ICAwIFVudXNlZAogICAgICBiSW50ZXJmYWNlUHJvdG9jb2wgICAgICAwIEZ1bGwgc3BlZWQgKG9y IHJvb3QpIGh1YgogICAgICBpSW50ZXJmYWNlICAgICAgICAgICAgICAwIAogICAgICBFbmRwb2lu dCBEZXNjcmlwdG9yOgogICAgICAgIGJMZW5ndGggICAgICAgICAgICAgICAgIDcKICAgICAgICBi RGVzY3JpcHRvclR5cGUgICAgICAgICA1CiAgICAgICAgYkVuZHBvaW50QWRkcmVzcyAgICAgMHg4 MSAgRVAgMSBJTgogICAgICAgIGJtQXR0cmlidXRlcyAgICAgICAgICAgIDMKICAgICAgICAgIFRy YW5zZmVyIFR5cGUgICAgICAgICAgICBJbnRlcnJ1cHQKICAgICAgICAgIFN5bmNoIFR5cGUgICAg ICAgICAgICAgICBOb25lCiAgICAgICAgICBVc2FnZSBUeXBlICAgICAgICAgICAgICAgRGF0YQog ICAgICAgIHdNYXhQYWNrZXRTaXplICAgICAweDAwMjAgIDF4IDMyIGJ5dGVzCiAgICAgICAgYklu dGVydmFsICAgICAgICAgICAgIDI1NQpIdWIgRGVzY3JpcHRvcjoKICBiTGVuZ3RoICAgICAgICAg ICAgICAgOQogIGJEZXNjcmlwdG9yVHlwZSAgICAgIDQxCiAgbk5iclBvcnRzICAgICAgICAgICAg IDUKICB3SHViQ2hhcmFjdGVyaXN0aWMgMHgwMDAwCiAgICBHYW5nZWQgcG93ZXIgc3dpdGNoaW5n CiAgICBHYW5nZWQgb3ZlcmN1cnJlbnQgcHJvdGVjdGlvbgogIGJQd3JPbjJQd3JHb29kICAgICAg ICAyICogMiBtaWxsaSBzZWNvbmRzCiAgYkh1YkNvbnRyQ3VycmVudCAgICAgIDAgbWlsbGkgQW1w ZXJlCiAgRGV2aWNlUmVtb3ZhYmxlICAgIDB4MDAKICBQb3J0UHdyQ3RybE1hc2sgICAgMHgwMAog SHViIFBvcnQgU3RhdHVzOgogICBQb3J0IDE6IDAwMDAuMDEwMCBwb3dlcgogICBQb3J0IDI6IDAw MDAuMDEwMCBwb3dlcgogICBQb3J0IDM6IDAwMDAuMDEwMCBwb3dlcgogICBQb3J0IDQ6IDAwMDAu MDMwMyBsb3dzcGVlZCBwb3dlciBlbmFibGUgY29ubmVjdAogICBQb3J0IDU6IDAwMDAuMDMwMyBs b3dzcGVlZCBwb3dlciBlbmFibGUgY29ubmVjdApEZXZpY2UgU3RhdHVzOiAgICAgMHgwMDAxCiAg U2VsZiBQb3dlcmVkCg== ------==--bound.487476.omv3tj5zwvd3plqc.iva.yp-c.yandex.net-- From nobody Mon Feb 26 22:22:36 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkFVP0XD3z5BnhG; Mon, 26 Feb 2024 22:22:37 +0000 (UTC) (envelope-from cperciva@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkFVN5GFTz4Nhh; Mon, 26 Feb 2024 22:22:36 +0000 (UTC) (envelope-from cperciva@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708986156; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc; bh=/RS2BSXhpCKY2iaxxCyWeuYe0NPN8nkJgnkHLPIKe3Q=; b=SVO5BAcsxt0YJeNWVfrKgNgnf7toOvoanNBy9bw4pROum6hsn4CtDUagZrEmyPPdqY7g8l 256NImJUWefR983IsLLFe7uSqGYklxeoKhYyeoVQ0tUKHxuK12ICHLzKo1lK00fKvfMTAa 2ViKa8mfpwlXG9gzx+T/o2P2xTv5PkusirrIuufeHQv9xLCxoRCsLGarHTa8QPtSLXT4W2 VMEjHCKJ8TsUAomeKXAY9SPrh5VYrmG9WLW2wLA8lciLM8DHlcZIat1y7cvgcVxchhB3oT 5G/LKMbQUaiM9tksEVfa+9MUfgWAFNGc8DH6G8+f253Z86VveBWP6KJeEaSTYA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708986156; a=rsa-sha256; cv=none; b=rUXbDh1NGKFatRlkDJEftIsh6qU1bFGfWE1/hbnBvkfM7z/w2dicKKcktNsSv34sTc4/jm PICvppkjpTv76QB44Hbp83tRht4rRY4YlGdy37ySVYX7zQLsp1mw28DyREN9qCuCZqBUeD oU+7/oWeofaRFNYQz3d6NYlt6shhxjIFMYAc8rru3yGSvX/5GVc+u/avLYvPZIKI/yI1tK 7a6jjQ156xTsclIRbo21MGL2PcFcZauB55cWjd79oUaS1dlY1TXPwgF9E2Oc3NzIZ/6gt+ NxZURfmOrl3o+mlIKj2s6T7uFbJesYcUGIuZ9/TO42NWokq21KNbftZ4MsmvIQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708986156; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc; bh=/RS2BSXhpCKY2iaxxCyWeuYe0NPN8nkJgnkHLPIKe3Q=; b=NnFU/dVdpZDYxpdGwNAW6JWB7dkOUCOJNCwnR9mLl4Rjyeq35pA5zBOkIBMJfMc31MaoQp RkR6dCUm7oT4y66PFjna4GbnFDTGT6xsb2kZjW17KAEu7gPkz3MAeRJNxi/nsfUo4JBJ39 gDx4o/wQnZSgzGcCN/43triFDyDb31/KS0Y2DLhFqctg9U3mlHku9slrNquTaiYKdu6ilD AP68Rjk0TH9+a5Hljv2OqRI6mQohKyyeA6pYed3kDTpMlSONgYwvLUna65ISW5Ee0FRYOH jwAekHTJrlkY9NtT3u7gkPlTB6hydPQCe+qMnCjdsX392uBYcoMUx8rBJitf8Q== Received: by freefall.freebsd.org (Postfix, from userid 1002) id 9ADA91BC72; Mon, 26 Feb 2024 22:22:36 +0000 (UTC) To: freebsd-snapshots@FreeBSD.org, freebsd-stable@FreeBSD.org Cc: FreeBSD Release Engineering Team Reply-To: FreeBSD Release Engineering Team Subject: FreeBSD 13.3-RC1 Now Available Message-Id: <20240226222236.9ADA91BC72@freefall.freebsd.org> Date: Mon, 26 Feb 2024 22:22:36 +0000 (UTC) From: Colin Percival List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 The first RC build of the 13.3-RELEASE release cycle is now available. Installation images are available for: o 13.3-RC1 amd64 GENERIC o 13.3-RC1 i386 GENERIC o 13.3-RC1 powerpc GENERIC o 13.3-RC1 powerpc64 GENERIC64 o 13.3-RC1 powerpc64le GENERIC64LE o 13.3-RC1 powerpcspe MPC85XXSPE o 13.3-RC1 armv6 RPI-B o 13.3-RC1 armv7 GENERICSD o 13.3-RC1 aarch64 GENERIC o 13.3-RC1 aarch64 RPI o 13.3-RC1 aarch64 PINE64 o 13.3-RC1 aarch64 PINE64-LTS o 13.3-RC1 aarch64 PINEBOOK o 13.3-RC1 aarch64 ROCK64 o 13.3-RC1 aarch64 ROCKPRO64 o 13.3-RC1 riscv64 GENERIC o 13.3-RC1 riscv64 GENERICSD Note regarding arm SD card images: For convenience for those without console access to the system, a freebsd user with a password of freebsd is available by default for ssh(1) access. Additionally, the root user password is set to root. It is strongly recommended to change the password for both users after gaining access to the system. Installer images and memory stick images are available here: https://download.freebsd.org/releases/ISO-IMAGES/13.3/ The image checksums follow at the end of this e-mail. If you notice problems you can report them through the Bugzilla PR system or on the -stable mailing list. If you would like to use Git to do a source based update of an existing system, use the "releng/13.3" branch. A summary of changes since 13.3-BETA3 includes: o Several wifi stability fixes. o A change to the order in which ACPI devices reserve resources. o A kernel panic fix when smartpqi(4) devices are removed. o A kernel panic fix affecting AMD microcode updates. o A fix to graid on Promise RAID1 with 4+ disks. o Fixes to filesystem corruption on msdosfs. o Fixes relating to ethernet packet alignment on vtnet(4). o A bug fix in the time zone information parser. o Several updates to Heimdal. o Unbound update to 1.19.1. o Expat update to 2.6.0. o Zlib update to 1.3.1. o irdma(4) update to 1.2.36-k. A list of changes since 13.2 is available in the releng/13.3 release notes: https://www.freebsd.org/releases/13.3R/relnotes/ Please note, the release notes page is not yet complete, and will be updated on an ongoing basis as the 13.3-RELEASE cycle progresses. === Virtual Machine Disk Images === VM disk images are available for the amd64, i386, aarch64, and riscv64 architectures. Disk images may be downloaded from the following URL (or any of the FreeBSD download mirrors): https://download.freebsd.org/releases/VM-IMAGES/13.3-RC1/ BASIC-CI images can be found at: https://download.freebsd.org/releases/CI-IMAGES/13.3-RC1/ The partition layout is: ~ 16 kB - freebsd-boot GPT partition type (bootfs GPT label) ~ 1 GB - freebsd-swap GPT partition type (swapfs GPT label) ~ 20 GB - freebsd-ufs GPT partition type (rootfs GPT label) The disk images are available in QCOW2, VHD, VMDK, and raw disk image formats. The image download size is approximately 135 MB and 165 MB respectively (amd64/i386), decompressing to a 21 GB sparse image. Note regarding arm64/aarch64 virtual machine images: a modified QEMU EFI loader file is needed for qemu-system-aarch64 to be able to boot the virtual machine images. See this page for more information: https://wiki.freebsd.org/arm64/QEMU To boot the VM image, run: % qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \ -bios QEMU_EFI.fd -serial telnet::4444,server -nographic \ -drive if=none,file=VMDISK,id=hd0 \ -device virtio-blk-device,drive=hd0 \ -device virtio-net-device,netdev=net0 \ -netdev user,id=net0 Be sure to replace "VMDISK" with the path to the virtual machine image. === Amazon EC2 AMI Images === FreeBSD/amd64 EC2 AMI IDs can be retrieved from the Systems Manager Parameter Store in each region using the keys: /aws/service/freebsd/amd64/base/ufs/13.3/RC1 FreeBSD/aarch64 EC2 AMI IDs can be retrieved from the Systems Manager Parameter Store in each region using the keys: /aws/service/freebsd/arm64/base/ufs/13.3/RC1 === Vagrant Images === FreeBSD/amd64 images are available on the Hashicorp Atlas site, and can be installed by running: % vagrant init freebsd/FreeBSD-13.3-RC1 % vagrant up === Upgrading === The freebsd-update(8) utility supports binary upgrades of amd64, i386, and aarch64 systems running earlier FreeBSD releases. Systems running earlier FreeBSD releases can upgrade by first installing any updates for the currently running release: # freebsd-update fetch # freebsd-update install and then downloading the new release: # freebsd-update upgrade -r 13.3-RC1 During this process, freebsd-update(8) may ask the user to help by merging some configuration files or by confirming that the automatically performed merging was done correctly. # freebsd-update install The system must be rebooted with the newly installed kernel before continuing. # shutdown -r now After rebooting, freebsd-update needs to be run again to install the new userland components: # freebsd-update install It is recommended to rebuild and install all applications if possible, especially if upgrading from an earlier FreeBSD release, for example, FreeBSD 12.x. Alternatively, the user can install misc/compat12x and other compatibility libraries, afterwards the system must be rebooted into the new userland: # shutdown -r now Finally, after rebooting, freebsd-update needs to be run again to remove stale files: # freebsd-update install == ISO CHECKSUMS == o 13.3-RC1 amd64 GENERIC: SHA512 (FreeBSD-13.3-RC1-amd64-bootonly.iso) = 802ba0f72a45965cfeed2c187e2622a72fe3d2d2eb501559d259008280b7bd4c6b430e72d1457f92086fed3120d4fcc54787a4d754f2ca3d60131d8d84fcd628 SHA512 (FreeBSD-13.3-RC1-amd64-bootonly.iso.xz) = eab3edc377d46151640c7a64287b012d0b556f3fd55198fcb50750a3c68f6f8873b8a957c8e3cd616ec5d1b013df935b8d889a30c2a53500021aec4dffcc4077 SHA512 (FreeBSD-13.3-RC1-amd64-disc1.iso) = 105ba6811a4340e3a25a5d19b9c3b13b79d80e50696d8c3966560474cc1ca07a81a5d7ab342751cd23174e856adfb18b86c27c8451a32e8652cfcf4b25856e4c SHA512 (FreeBSD-13.3-RC1-amd64-disc1.iso.xz) = cd164e8088dde546c2d29075dbe8084bf77c43a1a8d90636af1bd3888d2bb32d5ac299af6abf714f69b49f3a29070d6ed6040d0489d8ba31b53902a0a15dabfe SHA512 (FreeBSD-13.3-RC1-amd64-dvd1.iso) = 78ae63440fa9d3f9b4ef3e605425cd32d179dd545910e8274752b65e334dacd549d2d81953ff81f7b7c6818a4b911b349936b791a50e81f0156f90c71c7ac900 SHA512 (FreeBSD-13.3-RC1-amd64-dvd1.iso.xz) = 682f9dc46e6ad4db9194fbf8ce336a315b8f8d5256e87909256c7c388d3344b5b1387bbf30063668cd8917a890fef63fb5d99a8cafe2d0175bdc0cd5fc43084f SHA512 (FreeBSD-13.3-RC1-amd64-memstick.img) = c8a8c124a07e9ee37c5ec3151702cbfb2efe63bb271e667eac921b15627b6ee7241d6fa49d83e2676863d6f3b93a0816b6f191dd9c13560a82cd73800d49ac43 SHA512 (FreeBSD-13.3-RC1-amd64-memstick.img.xz) = 2b62cbc588b3d7db3f5d2a3020e363c4820983a497ad20087e413257f65b65c340ea2566bb5905a8808f3237ae0bf4bf975a5d569b3ab29acad9ff3994cd802e SHA512 (FreeBSD-13.3-RC1-amd64-mini-memstick.img) = bd3d8cbcb66b29f5163f6c62f97205bc5edcc2153bc5f88433b5bec0afb85de54d1a3f758f7f089d24cea287402887762db8615294aa398f37c6fbc74034fb02 SHA512 (FreeBSD-13.3-RC1-amd64-mini-memstick.img.xz) = 7929cee28e6f2d17e12de4d8f968d1a994cf11d4ce697b28160496cee7fcbf73c039642a51e21edabb21c931a53068aba1e9db3909f7b007ef0ef5ae3b24d613 SHA256 (FreeBSD-13.3-RC1-amd64-bootonly.iso) = 562ced9f1f002db4eb1d29dbb9aaa4416eb8946a11705adee6f2cc7d2fa8b130 SHA256 (FreeBSD-13.3-RC1-amd64-bootonly.iso.xz) = c41edde9784977168dc7a8299c46a00bea937a53667b2dcdf575c00ee8a0db22 SHA256 (FreeBSD-13.3-RC1-amd64-disc1.iso) = eb86508347189e66f2135b6ab70b452795374124b6d233b0d894ddce8dd84eaa SHA256 (FreeBSD-13.3-RC1-amd64-disc1.iso.xz) = dc04d85b80e82e932341cf1505060aebbb0ac70b1a595e97d82f142767ee0837 SHA256 (FreeBSD-13.3-RC1-amd64-dvd1.iso) = fe39b8a5716ffe2839042d806e7ac22c109b6d041db3eec83e1cbfb93b71f3f9 SHA256 (FreeBSD-13.3-RC1-amd64-dvd1.iso.xz) = bc231c978a5ab4c3ab1111ea7475e7bda3686977d50ffc852cb81b55f0c77080 SHA256 (FreeBSD-13.3-RC1-amd64-memstick.img) = 25da89cf26904890b9258e054bcac33a6ab66bc5f724a1b1d9f91b43e8a0d3e5 SHA256 (FreeBSD-13.3-RC1-amd64-memstick.img.xz) = 014f7fa91cfa4dbcf3525db2102dedb3b838c22592e90252c7db3bfb19f53ee0 SHA256 (FreeBSD-13.3-RC1-amd64-mini-memstick.img) = 27b5f15ed41150753851ea858e4b027351fd47cbd75a48302840d0989440cdde SHA256 (FreeBSD-13.3-RC1-amd64-mini-memstick.img.xz) = 5d13f24948496225c3c35e2a9e05933a0515ec8c74df324358973c259a3470ed o 13.3-RC1 i386 GENERIC: SHA512 (FreeBSD-13.3-RC1-i386-bootonly.iso) = 48e40b1f0cf73c9a7e9b4030c59cd4149e570bc81b0b074fce8cf7893d19ac4b0c56c2cd5205cd42116909eacf94832954fe639e492233fabf30afbd3ec054bb SHA512 (FreeBSD-13.3-RC1-i386-bootonly.iso.xz) = 6c9409d8842ee18126eed5090dee5c5b4d22d66c5293746d5be2a8336939ce82a908eb5ebde8d48ef64d242b9f1c12a16f28135e18d40bb94a78bd6c11737cf2 SHA512 (FreeBSD-13.3-RC1-i386-disc1.iso) = cac4c49a6ef35226c2940d4f345815c59d898c5d5d90ea82e80f0b258384cb5fa03f808637092a153161ed0233df0a344c2b1cc84d7fe15fac461caa582e778b SHA512 (FreeBSD-13.3-RC1-i386-disc1.iso.xz) = 620c1924fb414a525e1193ccea3ceaaf58133a4390b372d2e549d82dcd5dbf28d04fec80e476ff1db5b1719c9a2e36b1969907c08cccc0267c311260332c11f4 SHA512 (FreeBSD-13.3-RC1-i386-dvd1.iso) = ebdeca702c5ee9251b0dd43618e5cd0606b23476dffd4c093253a75dce59c469346e0ad24f65fa5ca26b34d594e27bb02c081f37903c5630096225f630eaf59b SHA512 (FreeBSD-13.3-RC1-i386-dvd1.iso.xz) = bbab9fb60d54cb41bc1d286252b88db8594c184889e9f518d2b4aca79d96a1f45aec7bd43284ebba10f2a97a91b1bd009c9e5489e9a51d3a41ed161110462fec SHA512 (FreeBSD-13.3-RC1-i386-memstick.img) = 2c911f19cdfe0d9ef8775d4db44920a525553babf0283f2e483e0acc65a09a1ee639013285f002c1df5aa16d1ee93717165998996a32e793a6b99fb4fd8769f2 SHA512 (FreeBSD-13.3-RC1-i386-memstick.img.xz) = fa46c4bfe77ac939ae58029c223e2290a56eb5fb9a99ab370d2387f2eef0168401c324e9277ec7ba190108fca58dbc6d7fa8d3faf5c1e27975ecf2fd9239901a SHA512 (FreeBSD-13.3-RC1-i386-mini-memstick.img) = 029c315bdee7177a05ef064066864e285da758671616bdf087165037b220e75c82a49a6d73867142d114954c9d3aca1f42193ae52141b17f918d29f7c2c3f601 SHA512 (FreeBSD-13.3-RC1-i386-mini-memstick.img.xz) = 9f7ecdfa384064d25d0e58adb305bfa18b22af3c68f62bbd5ddd50450ab0f55eb30b9371ae684220fccc9d55afd6ae385bee692da64673fc49f708f6508d4a8d SHA256 (FreeBSD-13.3-RC1-i386-bootonly.iso) = 8b50a3517931ce9e7cb696103b907da81ab4a53a13833a208f1b5291908b726a SHA256 (FreeBSD-13.3-RC1-i386-bootonly.iso.xz) = a470ae32bbede7bb736f5462d36279347c756e4cd4958f601c4558bd743ed047 SHA256 (FreeBSD-13.3-RC1-i386-disc1.iso) = 7d87778b01e48e05504c83e5c30524fe9f4f78a116419036b43e69867a20eac0 SHA256 (FreeBSD-13.3-RC1-i386-disc1.iso.xz) = acb4c594b8b613c7c594b3495cc3c3571e440a5aa954d7e4bb5982769d57d96c SHA256 (FreeBSD-13.3-RC1-i386-dvd1.iso) = 73b85e8de0117324ee1b41219442c74ef35799bcb33c3e7a2ada89205af46b5d SHA256 (FreeBSD-13.3-RC1-i386-dvd1.iso.xz) = 4bdbd65b78c5f8812e955e4b52a012d7dc18e9437e18f2f7139103d2fd7c2c4c SHA256 (FreeBSD-13.3-RC1-i386-memstick.img) = 96fe939004f486ed6f6383f0e0957c1f6163c80e71c6dd093f2643b870564cff SHA256 (FreeBSD-13.3-RC1-i386-memstick.img.xz) = 6441f1d9acaa7c27b7a6f720cd2b2d124db01dfe80d08e40206d65d43a8cfe70 SHA256 (FreeBSD-13.3-RC1-i386-mini-memstick.img) = 892dec45dc00e857266ed12074c20fb247b8953af5a22c0a5a6da1f01510b521 SHA256 (FreeBSD-13.3-RC1-i386-mini-memstick.img.xz) = cda48af09ae073a72171a3041c8f36d8d587c1f3dbf5b4bf8db6b0acf5b936ae o 13.3-RC1 powerpc GENERIC: SHA512 (FreeBSD-13.3-RC1-powerpc-bootonly.iso) = 92d2daded32b02e7279fbceb78bd39f55aa38a5014599ca48c27a06eb73a1babf0f2d467a70ae0f269a9feabbe5a0f292a9a3013e3fc9bf049c7514c8f1fd4be SHA512 (FreeBSD-13.3-RC1-powerpc-bootonly.iso.xz) = 84877e2c87ed165886db94e60a88e526c489263212197480019f7a4d90620c142f6fdc71f0aa09157b026253e76604260c55f0caaf05324540fe17b54cc0eef1 SHA512 (FreeBSD-13.3-RC1-powerpc-disc1.iso) = 4930daececa131f7842b17c7a747015effcf59864bfe52c1f4bc64d8002efc512151b6951121e09fe5a40a8c5d1519765141c15030c9c8057aff5c55b4a4c5a2 SHA512 (FreeBSD-13.3-RC1-powerpc-disc1.iso.xz) = dd1e9b267e6339071be6dcf21d2eb3a4963cd6655eaba67c0d494b1da2f9caadf26e8787b65b89a43018d3b657eebe34ec8c848904d42eb82ca5ebe21e6b8254 SHA512 (FreeBSD-13.3-RC1-powerpc-dvd1.iso) = 6286cdf4befbdf1e440b2aa2aad9329e9669eb364a67270ed4d0903be6fef0c7aa42d2f4656ea726e923e3fb4bab4d927bbe412786bf35c22686441b3ad86998 SHA512 (FreeBSD-13.3-RC1-powerpc-dvd1.iso.xz) = bb4e0e380def12674a12880e9d3160b07a851c2d9b20218b97a81e6bcc157eb7af3112aa0bf8f3d3e5469491543969ccf9f2b9133c15fd5f4513accb786b9b30 SHA256 (FreeBSD-13.3-RC1-powerpc-bootonly.iso) = 007c6e9ae9b737cb872265ec1a52999eaf3fee9ee68fbce6a53014df5e5b8e34 SHA256 (FreeBSD-13.3-RC1-powerpc-bootonly.iso.xz) = 573a53bb867f5d1aec007299593d0e30d89f7cc7b0582563ab70433d7dbc6202 SHA256 (FreeBSD-13.3-RC1-powerpc-disc1.iso) = 6aacdc6bcca34c6b9ffa0ca03a4f0b9effc1182e1498487c86bc7366a3d67c4f SHA256 (FreeBSD-13.3-RC1-powerpc-disc1.iso.xz) = 383186511ba791ebd3b1b88515358f84ffe541a2fc87124a5d002b5cf917cceb SHA256 (FreeBSD-13.3-RC1-powerpc-dvd1.iso) = 4927ea6fae4ed027bd5c624eb8a8a7f9affa7f0a75ffecf3ba7b421561f55627 SHA256 (FreeBSD-13.3-RC1-powerpc-dvd1.iso.xz) = e1f62576906e5030604ae4b4ae14e6c0cdedcbb12bad9f1ef84836a84b1b3d59 o 13.3-RC1 powerpc64 GENERIC64: SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64-bootonly.iso) = 4519a20af0ee36d677292536bcead5181589ca8e9afdbc2a29b8523ca5077477a159f8ac6a7a43099c0051febd4705b1e24cb829aa6f47780be131df7d738fe9 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64-bootonly.iso.xz) = 39026d3aa2ed1e90578637e9c1a776181e8ded208a9bb61597534e3b653d52299913b457091c309c78138b6b7f98049da53ec004986a40c38c017aa3a5bf2c4e SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64-disc1.iso) = 5c7b1e459fb7780ebb31d4f4b44c489cb75978df25e39c2b351fddc39a68a8f7cd4507c10dad38120b5260f0314c868dfa5842bb7ec3022c983b39c59032ea60 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64-disc1.iso.xz) = 881a123c33b2c8a0e5a56aeeac165e77e79825946c0a27d010963eec8bae05d19bca76a5ed1eaee05b15bd547c3b223c0bb7e8091693f103f12e6d34370e0782 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64-dvd1.iso) = b4f03210adcdee4d9cd298b54478b8c4a468880e154fa71cdaf758ce579b3b649d2599bf882f179ac6dc28195204a5763f466c332817ff759e8f03acb6f8ce0a SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64-dvd1.iso.xz) = 153f6d9920cfc3ef3f321af3c2ebbf4ebda364e60ddf18c7a250bb28cbc606de6522c52ecdd6b56e9b681908f66c068c8204092909cef6c822803d28d5bef773 SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64-bootonly.iso) = 4c5aa1d385abf92985b6f42a2fa2e5f32a3259bf3121a16679bbb1021697ce7b SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64-bootonly.iso.xz) = 2e54f6a18c866526de0ba34acd8dde36b94bdb1ce04c9498e1b2d31ba75ba16c SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64-disc1.iso) = d67dec41708e58ed2fe1c00324940adff5226f9a4d6779bb191d91cb2c8a2a8e SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64-disc1.iso.xz) = 084322fc892d30220d674877518cf683690ca8fd7d47084cbd1cb9259b629614 SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64-dvd1.iso) = 5569dfa01432357f05c89bf39ab29a5330da1615761059490cae51748cd9f48e SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64-dvd1.iso.xz) = 853724d4700719e15d9ea25097e7028aa0a0a6e191c9e89445b02556c00ffd42 o 13.3-RC1 powerpc64le GENERIC64LE: SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64le-bootonly.iso) = 6845e4d5029dcd16545347518fcc35945397ac8c8fec5a091b7babc2e08117fab3cc8aa5e9c30387c0da1bb06912b20b8bdaeb84d66b81da39fdb9fdacc24e72 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64le-bootonly.iso.xz) = 05b7848b057abc19c50495c5582c3992cc108e8f9ef63a7716e4586659e5e49361928ba64a93a35e12115f37a5bd75d7a09dadb697cef344e61242677bdb8820 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64le-disc1.iso) = dbd645afacabdd8366e8f646173c45b404961af753e8d2b9cabde0e8d0a1bd3da391d97d25d920571783aaefdc55b54038652e1a126848d709b8f7ece3b7965b SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64le-disc1.iso.xz) = a05e6b78f4a65329b919bbe759fc06f8be19f56b8c32c30a4e2ccd12c481d7ec17ee6b2cac31c197ec29bb6f844873c1c93d3673950aa80b6be8416d18472cc0 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64le-dvd1.iso) = d2527db9a0af581116a10e6056fd7a38436a8f0ea923f9a557d6c6a02a68d4733149ade014644f8348300182becf2b86f0aeddf97aa5deafd603ab86154fb419 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpc64le-dvd1.iso.xz) = 7a52de1ece5bb3a4c3b3916b32b0c5470789dbfd5b20d89902df0d37def6d3f842c1515e33660894edba3fcc4eda3829aa41f6677abc5102969f98488aedfb2f SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64le-bootonly.iso) = 0cc175a1d81151be979f2038417078bd4c95aabc03faac60b99e116d9a0fabc0 SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64le-bootonly.iso.xz) = a70f10baa4a6e85eae7581fe44d71be9ee3179cf68e28ea0d940c8d8ffb4dbbb SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64le-disc1.iso) = ae4b9e08794fa28f37e8290122e2101c68c557f5c8d4c16473594d0d10d5f041 SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64le-disc1.iso.xz) = f27bb0bf693a4957a266a59d31ba4dfeef28bc51b44b238859abcf68a6bb9594 SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64le-dvd1.iso) = 3f0b774d0a59eb028373bc9cd0291e1c36386432b32a050c6694bbf0ff761d44 SHA256 (FreeBSD-13.3-RC1-powerpc-powerpc64le-dvd1.iso.xz) = fed90d61ba29fae5f802271c49133c91e455c9220484b8f5491633f394926c60 o 13.3-RC1 powerpcspe MPC85XXSPE: SHA512 (FreeBSD-13.3-RC1-powerpc-powerpcspe-bootonly.iso) = 615469b879e0ad29ee77d67e8a571e32450c5f84e8e8a4469f27df377b8875554aded4b954bc951642d851035945ed959f94d61271644d296e3405b793331330 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpcspe-bootonly.iso.xz) = 34a623b0320c2a27441fc45b46c80098dabedcc327105469199ee8270d32783accc03d587673b6e88335ae9198ed343a03eadb1305fe422a13bc09c90c2636d0 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpcspe-disc1.iso) = 0b674dd773d277ef29766a63d99b19f87710bcf758b956cc6e689b0a5e776f782f08a8c3549d724f7c28d2e1ffee4d77772fc6d3e42ac8a8113b498fc3f64aa7 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpcspe-disc1.iso.xz) = 1db82922f433abeb2681d0cea2a38feaf1d20e08e3c8eadc85c6a649da69c91bc6cab1f0d847127f4c511a58a00e4abd437b861d97ab94a76b46da586be7fbf9 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpcspe-dvd1.iso) = 7a5acbe5ad77ad5f618828719bec8b070c6a564e25490ecb5651775b1978addba5e6971c59951bcb7897bc1c91e3abd4dcfeea53d1bd137a8ef7f127ae38abf1 SHA512 (FreeBSD-13.3-RC1-powerpc-powerpcspe-dvd1.iso.xz) = c75f1fa32530b6dd5b879e7484bbbad3b4ce77f8e061abaf8910798bfdce3fefaeb9db61a378c400fede2a4a80ca16b4fe3bc7e103483143a49c0d36bd33d4fe SHA256 (FreeBSD-13.3-RC1-powerpc-powerpcspe-bootonly.iso) = adff35f0e666b3fcaeb7abce8ecdf1576e06a1e6152c8f0bce9e885ed8a4d40a SHA256 (FreeBSD-13.3-RC1-powerpc-powerpcspe-bootonly.iso.xz) = 5b0ae4ecdd9622ff44e09c1af3e748f3ca2f933cbeaf9ba67a228cefe81cba97 SHA256 (FreeBSD-13.3-RC1-powerpc-powerpcspe-disc1.iso) = c5208381ff2b195e4dad77e8a0bde6ded6981b1aa64d97b4a0af641265c78d62 SHA256 (FreeBSD-13.3-RC1-powerpc-powerpcspe-disc1.iso.xz) = 406de2610a0a86425c4b786297e007ce40118c9cb83e827cac512599b5583f83 SHA256 (FreeBSD-13.3-RC1-powerpc-powerpcspe-dvd1.iso) = 0de59cb47c07b4c74fcf8a4631699185d54bffcad70d201864d744eac1febe8b SHA256 (FreeBSD-13.3-RC1-powerpc-powerpcspe-dvd1.iso.xz) = b0a8ae1c99c0c36114d06362687ca0f567bdd3b9bef04663d6969f639e6db8c7 o 13.3-RC1 armv6 RPI-B: SHA512 (FreeBSD-13.3-RC1-arm-armv6-RPI-B.img.xz) = 8e711929f2a929476e17f6578687799d962b5483aa22958ff93f3da52af17a1b499e04cabf656a1184cf833e8f69c9387568810ba9b5524eb2f02d7e02df891f SHA256 (FreeBSD-13.3-RC1-arm-armv6-RPI-B.img.xz) = 59cbde099ff9c5c4e880bf04c0e2dacd94cc3b45e2a8ea47db7384386444c773 o 13.3-RC1 armv7 GENERICSD: SHA512 (FreeBSD-13.3-RC1-arm-armv7-GENERICSD.img.xz) = 5d8d05557d2bfec5590b7c43e9e4988ab3fad961706085d16f50609e3ef73084d035750b6418d43b4c5a53e986cca1a518e9da36474d519990aad3db3a0ae0c8 SHA256 (FreeBSD-13.3-RC1-arm-armv7-GENERICSD.img.xz) = 7ce381b44fb36d40f39379cf903ff8f73fa49f05be09b4cf5d693dc15834e570 o 13.3-RC1 aarch64 GENERIC: SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-bootonly.iso) = 9f14202d91f53d7ec5cf8462c489e891a2643854c0e5d71fe01f20a0f7dbbbeddcda0b747e4c79e33d4543a5780472b9a631c808e53d638bdbb0e4a321855586 SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-bootonly.iso.xz) = aea9d6fe27995c6c4df12513a67e1e39f6382c36df89b7345512e00da1bf66933dd3516276ee67ea9258b32266c700b473b55c6e1b9d8674ec3ab2ae7c2dee29 SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-disc1.iso) = 35bc0889572569231af4f18bb6c9559d4838ed51bb98c3e53afe703e300d85502292e2cc95fa7baf173951dcfbc9027d652033aec98f82e952a699d14bfd7345 SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-disc1.iso.xz) = bbf9f22a40a5b71ccccc98adf062058ee49ef05b3e4facbfde070b6dac444efa99c08f7c9ac596ce526f532f6cf808eb4ff2df2162010eab1aefec3009160f74 SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-dvd1.iso) = d70b063c8f5528bb86d66b6afb21df25e80b829123be066e12bc57cfc1deb6a16167c1399aa47b7b526024035111724beaa042255a22ac415a933b9bc12f6fbb SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-dvd1.iso.xz) = db7673d74bc9bf5f404cabb18a77aca1091c6f6cb90edcdffbe6ef9b9e44739ac7c1d0fdce5068c8a6070a8a57773ce18757cff3c9a934bbe426c54147e28755 SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-memstick.img) = 955af99564883e39f3fae2b7f9a66fa9dbfc03f4da88d9900ff397164c7e36944de92fb613d406cb205d2bfa14e9a485c39dfd1150d2423fe25914484b3de9fa SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-memstick.img.xz) = 6bcdff43980420a53e1b6f63c54cc56d5d01cfec8e265d4599abcd8b2dd0de5819b3abc405f86f97fc7a6decb5deea37c37e99d00d385986d10d86409f2e9782 SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-mini-memstick.img) = 0b21352905a0dbce01c309bd5498d5f04892d6d2822322bb8d0863e1a4501cde156f3ec68cf3f0bd32ef834e3e8a814122464f6b35f9c839317ab5d1c5ba55ad SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-mini-memstick.img.xz) = bf7004762b66fc70e2a4c0ab634c94a01fd09d842b27873a96efb8462aee26bedd957638822afac3f10baf38f1e593c9f19f89069eea9ca6341ad07769d105be SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-bootonly.iso) = 780375260be12b7ca88ccb2d587001cd523c0902a48dd3cadf2a10195f1a731b SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-bootonly.iso.xz) = 6db1727cc5d3c36f9bfa4f8977e38e318079fb8250b9a8accf8ec30b245632c8 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-disc1.iso) = 7fed39c1120f5ae514bcf22f4993b09b54668d9985b53c61e8b4c3e87f3daa78 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-disc1.iso.xz) = 2f1ebaa5465c0547b0922c6d2bbfd7894b13ebc64c56105dff37f0a6d301e7d8 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-dvd1.iso) = 2877b41ec59ec4d1e633635bed36140232f558d4d329ab0e5e292d99b6d4f6f2 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-dvd1.iso.xz) = 46b0eecaa26af6a4748aa374e189023cb3f00295b929e94d15cd1118f7358622 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-memstick.img) = 4fc79233e3a22a1d03da1045f8582ef18508fa5159ea8678ec7cb57eb91aa4f8 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-memstick.img.xz) = 363e50cb02295f833b651e2068beeea871ccce0d94e2441d6cd4a8f3bed78a7d SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-mini-memstick.img) = 04b7a283d6bad7d3978ba7f4d7a64255a9f9519ecb4d372d5488190cb0071cee SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-mini-memstick.img.xz) = 30c4fbabff7a90720aa5c064f8072f76e65d3c646b40c53b4290263f46408e83 o 13.3-RC1 aarch64 RPI: SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-RPI.img.xz) = 2ae2a9436dfd21f9e0900512b6237a9b62eecd27efb2a7218c25d3faccc929286cbb6eb229bbec308e29a0e0789006bf74774efdb9ac6bfeaab7d478f8d50b87 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-RPI.img.xz) = f59e093a823355eecff04d8008e6ac18b36458c880895b583669240eaa3eebc6 o 13.3-RC1 aarch64 PINE64: SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-PINE64.img.xz) = b3d26aeb5d46d3b2347bb4a47628b3d2afa7d87d8705aa43e86d4f218a1b7f899732f3d9e618d3c604fbbd14f01abc3d6ba7befbd4e5adde495578434d2b8546 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-PINE64.img.xz) = 3e1557b732f76cf1a815141cfe3de39f8f90d55736546a57cb752510fbc6cbab o 13.3-RC1 aarch64 PINE64-LTS: SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-PINE64-LTS.img.xz) = dfb45ea24e3dfcf30d4524163936d2493d27aac080fe1f5e79cc9f33f7fc0698a82460bf614780bb898006cc8a79cf8fa91dc1455b66ba9baa8f8ea9c07fdf52 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-PINE64-LTS.img.xz) = 0959dfa6fcb35242130f8cdbf61bb3f554f0e15637f85c0c8ed9d72b664ae31f o 13.3-RC1 aarch64 PINEBOOK: SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-PINEBOOK.img.xz) = cba259c984f05ae2523a8f42c516640cebb0ca4de146ca0b31a7d3824397d0c4b673303902938403d44d2b582648eeafe994533e7904f815b8f3b0d5bba4d712 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-PINEBOOK.img.xz) = a9c65a71dda6e2746a5a83311489f1eb2a1882a838415911968b6354ea7e40dd o 13.3-RC1 aarch64 ROCK64: SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-ROCK64.img.xz) = cf4fc623394bd5dd9a9ad09f46db5ebfe142a1a091e78608f10bbfc52a6e528fdbbb276c608e98a066ae85df5f06c72efd3e96bb3fcaa08ecd53050bda745e76 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-ROCK64.img.xz) = 1de6e3e8410bcc684657df9a7e4f466a937a8871fe23d05bc6e56dc0c22b1251 o 13.3-RC1 aarch64 ROCKPRO64: SHA512 (FreeBSD-13.3-RC1-arm64-aarch64-ROCKPRO64.img.xz) = e0e3f559a74fc784adb19585ad894ea9c28c0b098e7221ee99836af8bfea0446babf1018719ebce75398fd06384ebad1bab6874e7ebf85c9f35949836a31b639 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64-ROCKPRO64.img.xz) = a8b0717a9470194fc298058ff12d4019cb158e0c9604b8c994db4c189986a07b o 13.3-RC1 riscv64 GENERIC: SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-bootonly.iso) = 3e775c1c0dd1409f413878fc52c968f1c379030fd03ecc3063da40ca46fa22f5e840024b67b94f40f3fb0268e0b575cf59e30398b6469a74a29b5afb724d13e4 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-bootonly.iso.xz) = 5193582be5716c5d6a9910b9aec8083488ef3a9cf69f73a017f76cc015ef3f0caeab0ab7653434b088cf5e8c539cd8dadd2f5cb50d689b402ea858052bfc8447 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-disc1.iso) = 95efac5ad3077539a1077fedf7c8d0cf7d04c98308dcc4252794467c0b080ad5c8817262dff67b8f4ac0698f5807acf7d3d17ec105f5460d927186dd439fdcb4 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-disc1.iso.xz) = a997e9d54e6ac5802eeb1ee817e6cb187ee266f6979b8a38fbbb7bfda13d64c605cd71732b7e92ddc86d0f138846a47c67919d517c8159e3b4d7f2df5fc825ad SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-dvd1.iso) = c81d92f48ce590c2be9d7139d5fc28ace59b39f682bc367f3828ca933fd2f6195b45eafa6f861b985df4082ad0e85b68c337cd7eec083fc9af81c003c49cc782 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-dvd1.iso.xz) = afc66570fe6d798771099024f78189f8fab3a6838f398f54c04fe920e06b3d02b4e278c151e79f2b086c92906747238442366ec546df4ee559424f284137c4a8 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-memstick.img) = 02e9c567ab8d5628bfcf03b6f7e00e3e4a687c4e849be6a3373432f7c8dd92efaffe14b8e3187cb48f11aa06e8879c270ddeebbcd29decf339f7683a04ad3717 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-memstick.img.xz) = be536aa0228465c602b2e8d83110711c5f0bf390a17549640fa4f0b173a778b60e59d73ecae21f8fac6b087ff960ca37a4c12ae1e3bd484f29bec4b3026c97ad SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-mini-memstick.img) = 37fcd30db5e1f15ce65aab7f345cc952a8396c2486bf59976a47bc3dee37bd2b73818b92c2ef436043a6ddbf041b44ed5ba712b43eaaad98070945a6868a0070 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-mini-memstick.img.xz) = 19807b49acf2cd78bb6b6f169a16024bcc03328233035eb882658ed4b462c420bd6bcbe777013231d6c49b4536d8b45208b12fb3849245e20fc82f4b62f80609 SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-bootonly.iso) = feb6e2cfe29ae17b7bdf932baa68a6e3d0da1c7c77b730f20a037043724c8f39 SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-bootonly.iso.xz) = 3d91ef7b77cb5d9af70f1b9fe10ea625ebe3418d391ac98e827c6c2c25c58eaf SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-disc1.iso) = b6fb47756c8a4904214c77c703777d655ca99746e6578d8142652d34f0d3de3e SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-disc1.iso.xz) = 87e7ed10f288c0b5c4516a19e5b5810f02417f2e053d3828c0ca0357c207a1e7 SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-dvd1.iso) = ac81452a134a23a745fbdf1ead3d2d299e82c93f563ef6bf49b15876df76c606 SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-dvd1.iso.xz) = d01bbd675c2a16b7e37a7b37b52d9610292fe722d7831b6b634bc158b15b8660 SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-memstick.img) = 228c098a2f84c1ef5af8f675ab7ad4ea455732605148210b1390e9898843d033 SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-memstick.img.xz) = 985b5be58ed26ae8202e614e1c92d784330a6cb00385ed30d777355b98b2551b SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-mini-memstick.img) = 4d560d4c569fb6c9b45e16c384bfc6f05ac62eedb92442b89d3985c1a43367aa SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-mini-memstick.img.xz) = 47c644dc6c10f13a63897585d2a692bda55e90ef12590010cc38f0cad93b9283 o 13.3-RC1 riscv64 GENERICSD: SHA512 (FreeBSD-13.3-RC1-riscv-riscv64-GENERICSD.img.xz) = d538be0a00129dff1753c1e3e7dfbe2b44d1f3bbcca4fcbe1cb785e9c2f93816efc29a09997187c9f7787f69820999101d7b6e2d18999002dcf2c1667710fb8e SHA256 (FreeBSD-13.3-RC1-riscv-riscv64-GENERICSD.img.xz) = 4f3e7d8cea0c02b54d612ca8d6392e630aac9ce66df1fbdab3362ad8e9fa7bff == VM IMAGE CHECKSUMS == o 13.3-RC1 amd64: SHA512 (FreeBSD-13.3-RC1-amd64.qcow2.xz) = 27d875b7a1b08cb086c552bcea42cbf2d48d6bd2010b1c2a84c49d9f09d56e8031fcdbae0c809ce29b11abbedb3d0258583e76da088f205704c9bdd05a27846a SHA512 (FreeBSD-13.3-RC1-amd64.raw.xz) = 349172a96d7f1196c4806f01c6787b382b9d8afc63654a42fd7728acd47f1d175a9011131ad654dc03e2491cb8b6c6628b43a5812766d7dc33d2049ae15cf72a SHA512 (FreeBSD-13.3-RC1-amd64.vhd.xz) = 2ba334022e0569512946271acd81fdbe08c7d0924243f3e66b43f1aa8d4cc06fa3b244c6c3d3ad6ca7b3dec9b5036ea6767dbded4f102806b30fcd95953a24dd SHA512 (FreeBSD-13.3-RC1-amd64.vmdk.xz) = 0624e2cd282469291b680da2f1d05edc662605170fc06f18ec71823e917e33081da8eafb27a91f8ddefc4e7fff208776604ab0dfaf5f19bf40e22d6334816ca9 SHA256 (FreeBSD-13.3-RC1-amd64.qcow2.xz) = 4a907ca51748d7c25e2ae6fa46c412ece39ca02363f0b1dd0063d0e73136e38b SHA256 (FreeBSD-13.3-RC1-amd64.raw.xz) = 61f30f8a1ddb4285f793e323aef2460464e9ecc928a3b2443a6de3fd28b09ae6 SHA256 (FreeBSD-13.3-RC1-amd64.vhd.xz) = 48ce223414375454ab2ef414a70400ef63ca86b8743fcd04e79ca28930dcf2cb SHA256 (FreeBSD-13.3-RC1-amd64.vmdk.xz) = ddd03b3b92043952eac0ac3cb8d39ad2a51ba3161726edda5c8618327b3d14a2 o 13.3-RC1 i386: SHA512 (FreeBSD-13.3-RC1-i386.qcow2.xz) = 56d4795e1108c838bdf9b65446307ab98b7458890ee5f2d9cd04333440cb0584cc7670eebdecd9fc08abc09acd85cadf2f156386082bece201e069ddd4d8b460 SHA512 (FreeBSD-13.3-RC1-i386.raw.xz) = 47da1a645e5f92287eab8e7605ee39aedf466658984d09282a93febab10d30cb3fc84d6c1718169a06eb21933dd3215804b9d102cdede946e321254ce45fd7a0 SHA512 (FreeBSD-13.3-RC1-i386.vhd.xz) = 64224817fa13bba385801e643d57ae38bb648bfa24e5f3bb20966051eca60c465bdc07035e8189b5596cf85672bca8414f0a4ae9bc4608fc78a0154937fc4cac SHA512 (FreeBSD-13.3-RC1-i386.vmdk.xz) = 4d2a4de41583d0718a4972c0b5494c07026ee92bee48c8ddf07362acebf2e0604cc321e6db44dacea9e0c86b360639782e1c77e621a61ca87190ebe2fefd06cf SHA256 (FreeBSD-13.3-RC1-i386.qcow2.xz) = 171fd50882404ca0640eb42ff3caa6b0349d8a2c9265695a9a776e42899d5532 SHA256 (FreeBSD-13.3-RC1-i386.raw.xz) = 9343dad2186fff202544f3306bd1b1dbdb00843831d7c22effc662c68564a0da SHA256 (FreeBSD-13.3-RC1-i386.vhd.xz) = 61771016586d57519183fd167704dd1cd5af3ad679e3255615fda4a68825c09d SHA256 (FreeBSD-13.3-RC1-i386.vmdk.xz) = 2ef0d1ff915b8061666cfa1d57646dc3511d00b8cb84b273c146530cefde949c o 13.3-RC1 aarch64: SHA512 (FreeBSD-13.3-RC1-arm64-aarch64.qcow2.xz) = 4415ca2100531e47ec4b019155a1e6ba4b281817b2053e5d938eccf359c1d55cd57dfe212c64e59621258dab60eb0b3a167f699f35d19fbdc8e89049217a7eb8 SHA512 (FreeBSD-13.3-RC1-arm64-aarch64.raw.xz) = 14409785490700d394df34b9d0983c8131e396a40f98186e973330cdca64b63241dafb4102cf46cc62e41d3b7f2c7a2215146d872e326992740920cfba50154b SHA512 (FreeBSD-13.3-RC1-arm64-aarch64.vhd.xz) = 7644afbe3f3e9c2e39e8911d5152aef01a9c7179d25c3c2db7e2e71229306c40d80abe21a0c3c1e0443a13462b397281a216d6ce8453a96b4a8588fa3350d47d SHA512 (FreeBSD-13.3-RC1-arm64-aarch64.vmdk.xz) = bf690be754ef2c010910a21cee5d241a4cc1c40877304245fb8573171c13ccd56400a671792dea421aeafd57645425842650abcaeadfb2ed68e746c6a92996c1 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64.qcow2.xz) = dd18bcda88e40b33907d7cf68f7a58b92c76c1008dc80dc3583c0475a24dd321 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64.raw.xz) = 52648792eefc359381d2b99f962c3b8ab718a58d89c1fa205e0f36438063837b SHA256 (FreeBSD-13.3-RC1-arm64-aarch64.vhd.xz) = b0184f671310f057dcfdcda0120f4e589fbf899d63fdbc197b1be7a278d8e320 SHA256 (FreeBSD-13.3-RC1-arm64-aarch64.vmdk.xz) = 32d57f97fc2c25dd24c085c8e203a518aba6ed010a061970dd8e221fd7f62169 o 13.3-RC1 riscv64: SHA512 (FreeBSD-13.3-RC1-riscv-riscv64.qcow2.xz) = 57cac4b1aa66e4cb4a45c8e164ac6aa143c14af5f3350740a16bc643b5145c03c5863b632cd01acf79661baf52f5efbc395268534d41833c0795e113a4359300 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64.raw.xz) = e4b2858c8378da9895399bef04799fa1a0370b6d0c415fd40b61359e0c0aeb71771b4463523877d75de6d5f86f6eedaf615b123c8ade8248f0ee9dcc23a8d4c6 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64.vhd.xz) = 4dd306e11d85280e147617daf52867a7192693bf180cc45d36b37f6de4f9b73e15f2c5100d832020acb85246e746e921d9d76ae9b0477c7cf67ee054b3179c68 SHA512 (FreeBSD-13.3-RC1-riscv-riscv64.vmdk.xz) = 27d894a4827e53d7e35b21e311c801720c3b97122e7fc59d933c6295f956f4e64c25027d45698279fbd2b88fea791806e17c3b23d7de07094a307f3fbf68e20c SHA256 (FreeBSD-13.3-RC1-riscv-riscv64.qcow2.xz) = 3992ef23fdd936b793bafdf6a1e45bcdaacf7413d668550d3144abaf9b705f91 SHA256 (FreeBSD-13.3-RC1-riscv-riscv64.raw.xz) = 6de95052d3df5ca7607f51952111cc5ff208b4091bc245e5baf20838c3fd740d SHA256 (FreeBSD-13.3-RC1-riscv-riscv64.vhd.xz) = 3d4882386d423ff05913fdc517e7ff07c8ad9360c4a05fd72dc330e97996cda6 SHA256 (FreeBSD-13.3-RC1-riscv-riscv64.vmdk.xz) = 66d50ce318bc1fba28aa540a0f3a9b44c2a89dc04b0888c03dec4104b9a61214 Love FreeBSD? Support this and future releases with a donation to the FreeBSD Foundation! https://www.freebsdfoundation.org/donate/ -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEglY7hNBiDtwN+4ZBOJfy4i5lrT8FAmXc9iEACgkQOJfy4i5l rT+QZQ//bAtmNz6Sw4Fl2B+HYvEw0LWyugyg0zhi+McoH8DPlp4jjEdo6SgypkB0 46TaCZOI7YNTgQf6NgqS94OWxw6DMjfcvl4wpIEpkoRirLhUunJxUNPV2niXhxId zqBC4G847vufyLchy4sHd2FJGvluHy6gP7FmIUOIrj/sRWEjSSXsT2GTaK7JQFdV zKAaqXJgwmRjuovUVVwyZmSSKsgGB1IsJ+HTbAwUmwk8nygEsZJ76M07+s3lFGVq gcAlWqvq6EMsZHB6jo8cI7uBNt7puPv6M+nRXKTez3+4iHWTQxtOWJUPPGVAri++ LMt7WNPKJTn1ST/0pQiNhjn3R3zaf65+j02KWez3A7w91NxDEbn4MiS+HwXQM5xs fkD/Xhiz/cDuM3Q2Xtsdgw7t/7eiqfmLn6X/a+BJd5z5xQKuppZzdwfH3NCoET5q d/zgQgDabKkazfRiVvxxgJgkKk3lhhQsjgr3Gp/ajY4JWetwumLUhcUa/ktUtbqy cBWiveESq9lYmg91CtsorCZN6xl+XBaYJKQQOuu3lD7hekndW5vlnRl1LXXdBpLt UapaRLQWDwxLioP7T4ONhCQ88Zc6viMYRwvi/0Gc51m9MhIrIl1CrHM+s53RyfqK /UKxqYEMT7Qz/dzFbOJMkvNjdXwJ1uGhyIZzaWHoHj+Iv2HXYXo= =4SfI -----END PGP SIGNATURE----- From nobody Tue Feb 27 00:24:54 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkJCd6FG7z5C1Gb for ; Tue, 27 Feb 2024 00:25:01 +0000 (UTC) (envelope-from mirror176@hotmail.com) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10olkn2040.outbound.protection.outlook.com [40.92.40.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkJCc5cc4z4f8y for ; Tue, 27 Feb 2024 00:25:00 +0000 (UTC) (envelope-from mirror176@hotmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=nPd4qMP7; dmarc=pass (policy=none) header.from=hotmail.com; spf=pass (mx1.freebsd.org: domain of mirror176@hotmail.com designates 40.92.40.40 as permitted sender) smtp.mailfrom=mirror176@hotmail.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P/J4kXKTFNrTqS0/TZ8HemwIqFEHcYf/ItC4Z/adgdMSZS237i0wpZN/h/2KFQkAhhsPe2swCfEjAuDRuVM5jidLBNbdDIncFBpMykLYsncEue4FiWE+b7PMHwVejL/Ygbai/Uy2ebnh7DVmaJoLsrnG2DyvAwDJRh2DgGI5lsOqVsw4poYAP+lT41Rm32rD0XECvGf4UfWfozjN1oIpl0WJw4XS5ah9+dHAXuT3dyVtNffbmni3/QwxOddMZ13tRvtJJDGj5dz1DU7jh/zqhGbbVL0acr97l3VLEU6nxmjCrHEdcrQ1LvmfnzB7QSfksj8xah64wGqmd/gYt3mRvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DqB+N9pmYKut47yfomN4A6lKBVvXXxtt5gP5D+zObAk=; b=GOQpBFNRDpSt/Sd1EzRs94wLQBwRvW9LRwNt/PBqYqIYV8dJOlLi82hs5qihLsfp3xHhR+SQOQe6zar+pLHkZIDnuOq02RsoPJ/+cbE5MV+fyiwYL8s9GCvkWSHOmj0AmWuammGJLM+gYUfiTx6aUQ5yevHD445FOz0+gyC3XEAQuvnAPs5CqovqM2n/TrtDGnWo42D+AcLlehdFScyfL3JsmsrpPJ6TebzRkrPHZ/RCqWgKkx53IrXRwTUN1/lPljopsH8jCklMW57FZ+61VdylrOYhzY2j8eiLC98OCiDSZM8t64uTlS7w7V4k9zE1G1+wOxIo4DnyWw6Wv/iQUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DqB+N9pmYKut47yfomN4A6lKBVvXXxtt5gP5D+zObAk=; b=nPd4qMP7QBXfOSeFV3Z9tbPRNAHfSI8n4fQR4sdI3Hn3UX33cnPASQ9+whbJ9hgSj7Qfbz+4s5ATgO+XDb1CVwTMt5kzcmaG2kyxhAE1n/l8jOKeIigf3PuvQzziOvkEJlmzgJ1gygFfS/uFc7E2XGbYbNq2Xsq2bvh1ANOybZGlE/PWsUsD42rHQ6IYxtuK+1qtOYhkaP2MEgZACvtehXmcHLgiC8+Nj1CIXcGfKRJvs2LHDJNwjWHEhiYBszH1LCTp4jCcvNuOKmgDKsbwUByyNgIDA4TFMd/2qrX9WwrCPKrAAOJV0EV/Bq2ZfCXQJYdtG2N1mwUPZb9AJztVrg== Received: from CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) by LV3PR11MB8601.namprd11.prod.outlook.com (2603:10b6:408:1b8::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.25; Tue, 27 Feb 2024 00:24:57 +0000 Received: from CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::e526:b74c:4798:1295]) by CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::e526:b74c:4798:1295%4]) with mapi id 15.20.7339.024; Tue, 27 Feb 2024 00:24:57 +0000 Message-ID: Date: Mon, 26 Feb 2024 17:24:54 -0700 User-Agent: Mozilla Thunderbird Content-Language: en-US From: "Edward Sanford Sutton, III" Subject: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-TMN: [sJlid3jv5AiQ4J4rHWS+vf6tfdCsAdko] X-ClientProxiedBy: MN2PR10CA0033.namprd10.prod.outlook.com (2603:10b6:208:120::46) To CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) X-Microsoft-Original-Message-ID: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PR11MB4770:EE_|LV3PR11MB8601:EE_ X-MS-Office365-Filtering-Correlation-Id: b8e0472a-efba-4eda-c8fe-08dc372a86eb X-MS-Exchange-SLBlob-MailProps: HIo0QlLMSrt0pceerHBvdukUdVR1IXBgbOFbwRijEAjjwmHyQqbymXoNOZRmc/wrFRXOLG2TTxkwXeScqZPR7Xwme6Lh/RyunH0hDHlV+BXygbE1kRR1WY3qfZdaGvK85j0fTtQpwrDXIWhR2Qon2LwB9JnKEhXdk8u8c2bCUoj/21TEf0/F9iKIaRu3EQMrrMEKK8Za6Gg3Znoj8XEmU97XZBr8Vuu2VBpVT1K4s6wqydNuVR/WxfAZrLC56GfweFPqD4SDmjn6+Ahf8tX7B+lxvkNXxO9yZr8MQJMezfIQ+cu0WyyAyhXoYoW/XnIP0ncUOExAHDyw0Tp8D8aZiuJ/xpRkFTaBDWwwkeUDAnql++lDNn8oCbrk7w+gayvnClL+5KAUHcbRZFWi5X1f9gEyVAgciDWixV8DYjDLnWAzOZp0uIMSWl9Hn3YodscBRzWr/lZ/16Baw2KgWrA15K3xXs8sns5PqNbR9Ib62flSxgNbXJCu0EHb9N5mQAySi/Ej5X248ObKp0KlmjB8tcS/OorOvAOlHGvA7G9BJc9yWqvY59O3KC2yMWHsWKCZanW3jWwpal1XtBovh33+BsK9WKDlkxw/9ZkpK1rjD9Ib3a99948keTCHNrKAnAV/u/wRZZIacr3IH9iI+LlwddzuqSULvnOuxlWc2wBlu9pFTA44YOOejDfrrcOEYa7Gjp0oIOiDTVWTg2m+cBhhjFa/COGDFKN16dORM0cxv/UfPmPX2pY7Yv3OXWoIsWpWJotNNavB1nIREsxHkPwFifnG/HlRZTIzUKudS6+BM+p9BpIscCwilMVGiA8WJ66HH3RJDgjJZu+PB5IpzoB9iQ== X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NPhXG84cqEn4Fxaavl1N3JcCXmHTnBRp+WywBzl1BLLJJk5iZc+EGqxWxovzyrhZVBFX2SjcaOIr6MKZSAXLjHC9JGfgb786HC7oOmxFQEs0BSJEj28WVD8iar6Xm0FRT0b87oqpuC2Pl7kqfnpZrKkop1eH9suCpf+w6+KBDJuWe7aoeN+8AjP13+YamNdjpxU5H9zMvhQlbVUrBDv5t+Sg52WBS+FD9T26aZBR2ZVU47Igzm5TM7wcoe39nx8Gx0REzqjwUCJQu4eenso9kdDOSlrN+ufV6V6K88o3nWl16NuLDkwhdIuiVrzXAN0tFCu25pLLQBFu8GAZY3fYGr/8LY13Ox1kWxvQdAibqHjl1h8ppanIRnCe6x7Vap3LSftbNpMHMBN75hWKl5WuVlspaFiaXky37mGx5EdKzknyYkHj+oedh4aSpvWVbdhOiipnCCrNT2Yl3vOytXsdDihTsEjYYmb9u0vecJf/k+r3XU9SeanH9DjrMvD6YP7VYAJDOpJpeXg7LMqVGgLsLJ209EvmfPhQZATnvKxMeihFVRTRIq08vZr1CatNmtvB+fL6kpbesRA5Yo9PuJPxu7SJJvPYXpdeJhV3cw74zpuY6aag04FcxH8Q42ynjgDFDFs45Wm0qX2wXSOoH2LrmQAW7sF7e9o244bqNyDJhZk= X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MjVQR045T0ZoaEJjbENjOUFhNGZvTjVNQ2k4Z25JM0w2WmNIS3RPNjAyYjFv?= =?utf-8?B?ajNjRi9kKzBPeE85dlRpV1VkRDdiZVdJa1ZGa3RaaEVubzZwK0xCQTVOVk9W?= =?utf-8?B?VE0vSEJtV2NnNmZXVGdOWGVIYTdMOVAreVg2OUgyNEdzbkNPcFFVUFIxLzJa?= =?utf-8?B?K3J6bWg5WHZxVWZRRHRhZDVrcnMrcU94bzh4Uy9jeGlWL3VUaEhCWnp0RmtD?= =?utf-8?B?N1ZqcUJEcy9YN2h4SFJnaVl4aVZ5c2srR2xwZkFvRWQ3YjQ4VGp5R0VZeWlW?= =?utf-8?B?NTk1azRTaXgxbCt1TVcxdXhKOFIwd25XMng0Y253eURMK2NuSkllRDNoQUpi?= =?utf-8?B?Zno5NWR1TFlndGlHbys3NGNZeTh6b2tJNm85TDZ5dEc4bDZiUzRHWW9TVG85?= =?utf-8?B?L25MOVI0UFdVQ0FhbzAvV1cyWEYzOE1jTWtjbitNaHlzSVZpTExtVlpIdFlq?= =?utf-8?B?VUZiVE9JMFhiMHlqVW1LcFErVnlsbDh6TTdpQ1R3ZlhPMkwycFhScmN1UVBn?= =?utf-8?B?NG9mOXZNOXZxbGFVbVRHM2wyUWhZSnE4dEVsYzZERUsxUXlHUVRJYW1VWFkw?= =?utf-8?B?emh0VlF1NFlnRUNFWS85Tlh1OURFZy83cVVGQWxxR0RrNWJpSHJ0ZnBRbjgy?= =?utf-8?B?Z29KTUFtQjgrUzVJMGJSUU9lenk1Y1VYcllCQ2QxQmhMYU4vVzUwSVEzRG5o?= =?utf-8?B?T3UvelBGZDlTSXNHK2kwQU9hRXpGRU0wVkJZS3UwNU1IZzhabGxBUDBCdGdQ?= =?utf-8?B?VUt5ZS9DMVJnd3E1VjkzZ2Qvbm1RMzhiVmNZdGE0aFdYNVhmVDc5VEpkakFN?= =?utf-8?B?dWUrVlB3bm9aY1FhMmFtUzRCcUNYYzBobnFCREdpOFoyZkd2eEpVWm9vWWFn?= =?utf-8?B?TnlXOGgwRXNpclpENE0rTEZqSmpXeUswSGZBRENSZFlzSWRpemlrdHF0Q04y?= =?utf-8?B?L2FzalZacnJHNVowZEJoRk02TUpueVFaUUd5ZVFUTkcrNUNNOVFPQmhiY3ZP?= =?utf-8?B?YzRlb21PNjNVK2g3dUg4RWZ3YTBwUndlTEFNNUZzdmU4YjJCV0FUdnJPemZM?= =?utf-8?B?azlpaVk3dWk5TVk2VUVoZ3djLzNxb2xtdHQ4ZStheWsyaXl2ZVpQVUJKR1N4?= =?utf-8?B?Q1BHYmozTG0vM215WkRvZVptREMyVGJpN2I4VlBlWTdjcW9FN0hITkczSzYw?= =?utf-8?B?YWtUdlhiVThrZncwNjdtNEN5Ly9XcEMzRmoraUxYNjVOL3pGQTFtSTZNUWJU?= =?utf-8?B?YlVSN0lIOXcrRXFLL3kveWVmRXcvcVNEandWUGJIeDB5NVBzMkpwYlpEUjhC?= =?utf-8?B?Sy9BUWxyZ0htYzB4dUVFWkVFQ3FhTkk3VlVuTmQxb05ZbWh5UDBNbnNkMTZG?= =?utf-8?B?WDZVb1N6eHU0OVYya1JHblBkZGs0dkFIZCtmTEpqYWM3N282aTFvaWZScExO?= =?utf-8?B?a1JYRUdLQjUvUlV4eFdVK3JoWjdZYlgwQmFmbHZMWVpTTFdLVnZ3RVo5eHRD?= =?utf-8?B?N0dFOTJmUlRMc2Y2YWZ5UElEV2RuMVhiN2pXV1ZiL2NCbEMzTWVINkpXaTZv?= =?utf-8?B?cVZQc2xUeitjQUVOaXFqUUxLeG00dlQzSlViVk1RdWU5TkViZWtMVVBaYUgy?= =?utf-8?Q?d7CgZEX9fC8S5IASl5qAVdqlG/YwCFp7aSuSTgm1SECY=3D?= X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-e8f36.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: b8e0472a-efba-4eda-c8fe-08dc372a86eb X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4770.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2024 00:24:57.5455 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR11MB8601 X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.49 / 15.00]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; R_SPF_ALLOW(-0.20)[+ip4:40.92.0.0/16]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_FROM(0.00)[hotmail.com]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[hotmail.com:+]; TO_DN_NONE(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.92.40.40:from]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US]; RCVD_IN_DNSWL_NONE(0.00)[40.92.40.40:from] X-Rspamd-Queue-Id: 4TkJCc5cc4z4f8y Currently trying to do port builds with poudriere-devel-3.4.99.20240122 on 13.3-STABLE FreeBSD 13.3-STABLE stable/13-n257396-134580c103b4 GENERIC amd64 and have had poor system responsiveness on this and a -STABLE that was likely at least 2 months before it with `/usr/bin/nice -n 18 /usr/sbin/idprio 31 poudriere bulk -J2:12 -j local -p local -f /root/installed-port-list -f /root/prime-origins` and also launching it with no nice change and just starting with 'idprio 31' (in case it would react different with the builtin idprio. It used to be that under just idprio 31 I could continue to use the machine during builds with minor impact on responsiveness with MAKE_JOBS_NUMBER=4 on a 2 core + hyperthreading i7 which lead to an oversaturation of 8 build processes across the 4 virtual cores yet stayed mostly smooth. Responsiveness seems to have strange effects of laggy mouse/keyboard input and thought I recall logs saying system couldn't keep up with the mouse communication, programs intermittently freezing/unfreezing, and even building becomes unstable with electron28 failing as runaway process after over an hour when killed as runaway process during extract phase. During the lag, programs seem to take many times as long to respond while also getting high cpu use during that time such as tmux switching between windows; ctrl b+ctrl+p getting almost 100% cpu core time for about 5 seconds added to it under top. blacklistd has take about 24 minutes of CPU time for < 3 days uptime per top while managing a usual slow bot break-in attempt on ssh. More recently looked and see top showing threads+system processes shows I have one core getting 100% cpu for kernel{arc_prune} which has 21.2 hours over a 2 hour 23 minute uptime. Previously I know I have seen higher system % time than I'd expected but not always sure when it is justified or not. I started looking to see if https://www.freebsd.org/security/advisories/FreeBSD-EN-23:18.openzfs.asc was available as a fix for 13 but it is not (and doesn't quite sound like it was supposed to apply to this issue). Would a kernel thread time at 100% cpu for only 1 core explain the system becoming unusually unresponsive? cron was stopped after last boot so shouldn't be throwing up any unexpected background work. System has 32GB ddr3 RAM with i7-3820 processor with only 2 enabled cores + hyperthreading in BIOS. Issue appears specifically when CPU load is high and idle% in top is 0.0% but it is only sometimes present under that condition. I usually use ccache and WTIH_META_MODE to try to speed up compiling base and ports and have zstd at best compression for packages in hopes of faster extraction at the tradeoff of more disk space. I haven't tried yet but considered trying OpenZFS from ports. Any suggestions of what else to look at or watch for? Thanks, Edward Sanford Sutton, III From nobody Tue Feb 27 01:04:22 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkK544JRwz5C4Rt for ; Tue, 27 Feb 2024 01:04:24 +0000 (UTC) (envelope-from leres@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkK543mwcz4jnF for ; Tue, 27 Feb 2024 01:04:24 +0000 (UTC) (envelope-from leres@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708995864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=HYxCX+4fuX8qECSMGHGxBlXRDSPQU76i8ieFk0qFjl8=; b=keb0m+LyKx82gwEZQme+eml4Lowg9jCaMiKG1Xlx9XENnRpKfpzJRWtv2i5StAsLsP+9Mk 0sy+5QmfZncKRPKAKelmxynrJ9vPJLEqB1nMvTasVcXaOxnfgSYDs54YjtSoKrIqfhsZ0q WxpY0rdbXLUQ17aaFbqhsDz5/BsW74/muD3Qx+PDp0HnIryhbCnEBVeGFVdNYyeA67Om/S 3uXQKe1A5ngrzBYeX7nNsQd/Od/t7UPCWQcO9Jp2gD/I/+6GG3jnJO4mJtMfYkPKmAZHn0 EsBkoHyExlu0JRV/bgjeUzpbC9g1KCkNNgaGJcDWwrtHxIioJI7TY5pKc2Fr1Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708995864; a=rsa-sha256; cv=none; b=imKjx7wzaWMoijM9eW2mHuabCSxSOHJi0+PIf/Lxp1IESGdmOU+LptwMSPCygShtsBk5Wi Dfg88hIDjKcybqrWFTqEVs6RzuqmTusjwBgADKUZCX5MLMtTFOoo8ANiFKqeTCowu4sl80 KapEPm+jhi33oZqBDqd4frlqME2eLDBlWfGzD8ZQsxJGEBkhezWVUilZc9JK8q7iU2HyZk WHt4kGy63jbb3ENCtNBlwKda+AIrc5FFqHAQjE0rqea0ITSZJQMlvlukms+KJ8kOxNqaiF TLXZR2HwiatVeoPBgPV6m5KIuUwz1yDOXkVRwSIVS04QLGIvsJ8PlDztKWu+8Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708995864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=HYxCX+4fuX8qECSMGHGxBlXRDSPQU76i8ieFk0qFjl8=; b=ZkDphyxPTJo/XKW7lETBK+ZWQuIIDA6guKLopV6X8eq69hFXw3lStQ2HU2D713sy1FFpVR lGD/sPuZ4wiERr/Njfikj58x8FwVhJ3WV2G2pJHjwWvLfgHsd2KPWwnV82yCs0uOmVxm0b G2HcJH2Pei/Xl24M5KRgFMk7HX362x3XEfgFKw8YfySid/tOjQ8gy66khsf1gMvdFHkhz7 iNjD8zIQJUMk+/eZv20uoRasIDTtLE9i/BBGGyBbCD7OTD4U2whtKIdQYzgA46JhLGlOop y5kq7uzzLWz/bkixH53sJ9ICbnCl36BbRr1MBA6p0tgZGEBHDSib98OlsGsuqQ== Received: from [IPV6:fd:1965::2] (unknown [IPv6:2600:1700:ab1b:6800:2e0:edff:fece:8f27]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: leres) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TkK541V7Mz1Pg2 for ; Tue, 27 Feb 2024 01:04:24 +0000 (UTC) (envelope-from leres@freebsd.org) Message-ID: <80a7ec56-0e9d-4f26-bbe1-427c8672045c@freebsd.org> Date: Mon, 26 Feb 2024 17:04:22 -0800 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Craig Leres Subject: net/libpfctl: needs to be updated for 13.3 To: freebsd-stable@freebsd.org Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit While building a bunch of ports under 13.3-RC1 I find that libpfctl does not (build). I freely admit I do not understand how this port works; I started to create a PR but I don't have the first idea on how the distfile is generated. Meanwhile 1136 other ports built for me without issue. Craig => libpfctl-13.3_2.tar.gz is not in /usr/ports/net/libpfctl/distinfo. => Either /usr/ports/net/libpfctl/distinfo is out of date, or => libpfctl-13.3_2.tar.gz is spelled incorrectly. *** Error code 1 From nobody Tue Feb 27 01:07:28 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkK8n4FXmz5C4hY for ; Tue, 27 Feb 2024 01:07:37 +0000 (UTC) (envelope-from mirror176@hotmail.com) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10olkn2104.outbound.protection.outlook.com [40.92.41.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkK8m372pz4kxf for ; Tue, 27 Feb 2024 01:07:36 +0000 (UTC) (envelope-from mirror176@hotmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=h2UeovvR; dmarc=pass (policy=none) header.from=hotmail.com; spf=pass (mx1.freebsd.org: domain of mirror176@hotmail.com designates 40.92.41.104 as permitted sender) smtp.mailfrom=mirror176@hotmail.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mjzbP867pyd7Mpz6/QRSwRMRw1RpftovCZRpUQ0JoOjHyWb1wHX/gXJyaCL9yKLWC0+03ekXeNeCgiBd302RDIWWXHtSYgE+fyTai+Umcfvhz3rRrnMfLV9zNXipVrUk0KYCWeNu+zSK077ALk3yUsYs7NUG1qaKS5UBSzlmZ5+DNhK5Gs1N2wRwhmlCqav4qywIh3z4OX87wfHTHbxjGQZV+R1YqEnNoxltPchTieMTQneCf4sFbq4r5RepH7S+YpS9Op30yPu+9re8lLOEZHy6vjX1wYHM7ThE8lq0H4Kcgw6wrQKSJ7MCRQ4Xe+c63ISDFbOIUhcQGHMpaSFGyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zoLvLpFfDb7fuuw0tTdAw8SE+1Bquj4cqXROIqIDhQg=; b=P8dmcJ/yHX3xCULDF2+9nsBtS5AO4gY3Q6J7Lkb9DGOAyQHF819avuqg27HhnA8oYFkUzHGJQeZj0eLD9WCts4HF1gHggJ5S72esMBaDGP8jwZi1Ptf8imkYAWzzLkjTjxov+4Pcnr9lw40AjjH+ovNIbSlriHGiGqe76NtDyZZLasHOp7bqN3vt1k4AfWNZotN+D8+Tw3Kt4IH+IJxU9SumS/c+wOMt1Ui/98/Cg95YrCIpvD+dwzW3JP1bm4TBr5ELX3LSAJfZ3KmTU+k2deVELSvarrNM6lXUXU+s/UrnSWpwLD/QiDyq0QQOQMH2i+uh61VGpt4YeIqYcLmxTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zoLvLpFfDb7fuuw0tTdAw8SE+1Bquj4cqXROIqIDhQg=; b=h2UeovvRSEMy1Ov5ovaZewkn0DdJydzSLJaTVDkt+1PWSa+qUqYtG0npinRbgnPOLLabtHxqQGJiIZLSjjNJQ5z8ySg9DckOwQGkT4ciWENrtZb3Puw5M5TFMGqcSD0VZVmfdqvExqegQOzOcyuIibSk91oPysfpEOFORQ74O6XDwzS/qRxbjHEeS+K+wvaRp3Mv7E4+YrptJ2qVFF+Zz4EsUXnHiKKmf+UegJLFrxtlX9GFctvSmCthZiFIDD6JQz66jkNQ0Rwe9l2TRxEOmY3T/VJhXC8bLV5SQbpKSmWDd200MspxUdNY1J+1NQKW5sSSXYTj0EfywHrJsZm5Rw== Received: from CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) by CY8PR11MB7265.namprd11.prod.outlook.com (2603:10b6:930:98::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.10; Tue, 27 Feb 2024 01:07:33 +0000 Received: from CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::e526:b74c:4798:1295]) by CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::e526:b74c:4798:1295%4]) with mapi id 15.20.7339.024; Tue, 27 Feb 2024 01:07:33 +0000 Message-ID: Date: Mon, 26 Feb 2024 18:07:28 -0700 User-Agent: Mozilla Thunderbird Subject: Re: net/libpfctl: needs to be updated for 13.3 Content-Language: en-US To: stable@freebsd.org References: <80a7ec56-0e9d-4f26-bbe1-427c8672045c@freebsd.org> From: "Edward Sanford Sutton, III" In-Reply-To: <80a7ec56-0e9d-4f26-bbe1-427c8672045c@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TMN: [BYkPkkpMU2cGLprrFX+iL4DwHxL+dooU] X-ClientProxiedBy: BLAPR03CA0179.namprd03.prod.outlook.com (2603:10b6:208:32f::33) To CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) X-Microsoft-Original-Message-ID: <3a4c4475-390c-4b67-a2a8-befadcd9dd03@hotmail.com> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PR11MB4770:EE_|CY8PR11MB7265:EE_ X-MS-Office365-Filtering-Correlation-Id: b1c2b966-9fa1-4508-5efc-08dc37307a19 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FGsr146kjrjBkkJdKUuX1wI1lXDqUno1XAPWriAdygluctHVk1+se5tf/ty5kQgCxpCpNk6ncQ3gP9J0zmSikP/LdEgvxwyIwsF0jBLnCSTAosNAfUdpHtGZRYwQOwOGJPO3nW/Pm7FZ7fp1KfjVk5aT85Cf40WLxyk6aC3RrARjF9C/Kj9d0KNByQli2wEteyWoC58pmrylVqBzXaVAFpFl/RzUZhk4P2CgF16gZS9FozLzIG9knnI407dAR2UqA7kMSY35V43yEDLve3aExuDbRWe4qsZfQBo+dyykNXHRVHPa4RE+8+zuOb4l9YWw9oEZ77Fz6ZqCVFvNQz9TO2/OWsYclCwc3KdTJQQYUFc9g4zwQLr0OVWGbX+kELJnhMZplLhpstzgqzFRBPUFuE1JVSkPips5CBWi4PQ4wjot+hr+4yZ/mZzOW9lqPO/mS1Ka8UMv/uCtQTvUHls55ntxbdtx/zdxKsT38N4g6ucwLLNfF5P/kqxXlUC95q/inR1GteEqVHA05WKY5MURxM5jp4SGfUzMYpczymjoI8SSAmcDAtDIe2DFR1AMRjR6 X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SmpZRkUzTS90MkhKSXZSZUMra2VvelpwUHFITWlyajNZUjlwc3lGV1FUL05N?= =?utf-8?B?Q1F4UmpPaUtyQ1ZvanM3c2NXTTJxc3NUdjluOVNiMkpJS2VPSzN3RFVMWlZi?= =?utf-8?B?am1iT29RZHNZUGJNQTUvQ0t2c2xjQnJ4SHlDbEllaEFjYjR6N2VvYzBObjYx?= =?utf-8?B?RUZOYTFOUmgvM08wcDRZTkxRYU9wOVg3RExyUHdOT0JsWlFBMnBCckFSTGZN?= =?utf-8?B?VnE0ZnJzMkpXemtwaFI5VVYzT1lXekNKeTBkeTUxMUFiVFBiQzRMYWxCWTVK?= =?utf-8?B?NldsYmRIYWJWUnErN2djTnVGSDNFS3VyTjRmeDVkelBZV0MwWlE4SWxwN1NT?= =?utf-8?B?YWNLNWpZbGlpc0hQNlZBaTFGbzZvODhubnhzeExhczZ3NCs2TlpCc3lCaVdJ?= =?utf-8?B?a1VPZHMyV0hZclBoRG1PcFI3ek5SZ2lpVkZrOFExK2dtT2hkN0Q3V2YrdEV1?= =?utf-8?B?QVVCSXduN0k3aEhwTlZXNDlnMWtWVG5sYlJQQlhUM0YxMHdjUW1mL0VaR05M?= =?utf-8?B?ZjlYdmNCSzFCYlpQVTVvM3pNdFF0S2FIR29wbXN5N01yZjN4TWJwdWN0NkVQ?= =?utf-8?B?TGlsenFBT2NYTTF6VzhDRWJ0TVdNTDlwSkcwSzd4RXlkTTROUEt2dlp5R2t1?= =?utf-8?B?Nk5oZFE1UFVnSVVSQ3lFNElUV0hXb253c1RiZk41b2pxaFQwdWVYMVpnNjAy?= =?utf-8?B?NXJxK0RWZk44WmpvUVF6YWRzRUdYeEg2OVludjVyNmRhS0VvQm1rVXd2bkZ4?= =?utf-8?B?UXA4Ry9JTFN0S2JBaHltM0RnRlB3RTVOM204OHRuYjRUbmZXdHBrYU5ZZWhY?= =?utf-8?B?K2hIZ3lzVDQ0Z042VlB6MlpuZ1QzeURyQzIwNXdVZjlwU0RyQTJ0UlpkNEUy?= =?utf-8?B?b3RPV2MwQWltb01iUVBKZlFCTXRrMElZQ3NmSUlDeE9kS2REYUhQZmttVUtN?= =?utf-8?B?cjN4cS9VSDdTaUVVejh3cUd5dlVhQndXRHB1Slk4RFJXbjFUUWY1dnVxWXdW?= =?utf-8?B?bnhtSjhJaU50Y0hUaFRuMGl0YTJaa3JFOVgvWExhUFM2WEZVSlloMmw4Q0Iw?= =?utf-8?B?RmJXVXUvZDlFMW1WMnpud3BURElYNzdFRGxUL2Y3TGFKSW5obkpobTdsWFZ6?= =?utf-8?B?SSt5K2VyaitWdGVXTmkwbzFXKzl5clMyR2RMcXpaNkVXRWhVMVY1OVcvM0h4?= =?utf-8?B?elR1MktKakJPd1hEb0J1M3lDSVh6eS9IK1J6ZFBoV3gzenpRSEFOVDdjVEcx?= =?utf-8?B?RjlkTG1lRy9SaERBZkMwSmNLM05rbG5xLzlydHhsd2tobWh4RWFJZnRlWXRH?= =?utf-8?B?L2lPbU1rYStjMHZiRVJ1eU5lOTJFYm42cEFjQlZ6OXlrUHBsVmZ1azQ5NlpL?= =?utf-8?B?SE9aKzV3RlBVdTAxNFVnUVU2RVZZUEVXY1NnUVg0cDZQK1g1ZllmRnBYUnVE?= =?utf-8?B?QWVWT3laN3p5Y3lCKzdPMHFnd1EyMGdNNGZGM2ZDT3hEVVkwRklZUzg0RlZB?= =?utf-8?B?QVJKLzF6SWpuUlJER1d5TEdRR0xjRzhoeDIwemhxZkFxczExdGd3ckt2UFZ5?= =?utf-8?B?SmVNVFB1bzczNTJaV2hlL2EwVkh0RkFnTXdoaDRFTjdGSXI1eWZ6ZW93bzds?= =?utf-8?Q?3OyYiqxCv/uPghHRdzN2hevEghLVz5Imc3/WCSmit49o=3D?= X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-e8f36.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: b1c2b966-9fa1-4508-5efc-08dc37307a19 X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4770.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2024 01:07:32.9466 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7265 X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.48 / 15.00]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; R_SPF_ALLOW(-0.20)[+ip4:40.92.0.0/16]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_FROM(0.00)[hotmail.com]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[hotmail.com:+]; TO_DN_NONE(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.92.41.104:from]; MLMMJ_DEST(0.00)[stable@freebsd.org]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US]; RCVD_IN_DNSWL_NONE(0.00)[40.92.41.104:from] X-Rspamd-Queue-Id: 4TkK8m372pz4kxf On 2/26/24 18:04, Craig Leres wrote: > While building a bunch of ports under 13.3-RC1 I find that libpfctl does > not (build). I freely admit I do not understand how this port works; I > started to create a PR but I don't have the first idea on how the > distfile is generated. `make makesum` or hand crafting it. If you update the port's Makefile to be a different version or use different/more distfile(s), you can tell it to update that file with this step. > Meanwhile 1136 other ports built for me without issue. > >         Craig > > => libpfctl-13.3_2.tar.gz is not in /usr/ports/net/libpfctl/distinfo. > => Either /usr/ports/net/libpfctl/distinfo is out of date, or > => libpfctl-13.3_2.tar.gz is spelled incorrectly. > *** Error code 1 > From nobody Tue Feb 27 01:19:13 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkKQB1NTRz5C5cS for ; Tue, 27 Feb 2024 01:19:14 +0000 (UTC) (envelope-from leres@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkKQB0w6rz4mvW; Tue, 27 Feb 2024 01:19:14 +0000 (UTC) (envelope-from leres@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708996754; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gZuSEC0XkXFwcYNwknnpwoG7sl6iWmfkB7gqgeAgx28=; b=H5x3OnzXtxBDyCQA+z5mtWQcVGe3XM2DQPeGrMr50ekWOWgBzMtsbq6IbUONK5hBtFw5EZ xItBaSZhtC7+S2vT/a+ZPzMOEFFDf//G/pabclez2O1c5mR+81XFpmHVOny0zqtuji5Y3e 5r9CRDrVWVE2yZeR5A+k3PoEC+DY66qpYxhx8F1LCrZrUj0gf+0OQ9cwXclcoAVbcerogF 5rgG+CeOXsh4jetqc82c692z1yj7htNrqlWDaPJRkc4iO6DjLoNKkWqUqxW99gIylS3GVX Ngc+Uhn2dzr14Uo8jnQiuRVn/vClYkq3bxJAGJGck9ZPMk/HPQZsaiNnizaTsA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708996754; a=rsa-sha256; cv=none; b=miLN71vnadrqJx1pkhjcvTr5sf2H9TPIBvWJEf/2cx23CKV6N8IyXPe1ONymoSuyClNTeq +uvi5GrsT9qB9JJdjVXAwSIJUW7EAGGqKByrcwj/8A52HEhmasC4LCQdqpjY45L6NEXEdY Y7d2ZEpDfuVS2oSsem9XAKVFQ6N7sl9FcPfNo1iTJtLEuP/cZt/IVokwzhaTR9xIWAgHDF yWaI8I1w1jkAIIukJX148jtKFHafJQq5HC3FoBQ6ikJJRVDrsZJd0MNQz5nDBTbaAbLdkg m37X+VJQndrW0FlN5Qz7injJrBop0F5fQuryjvGRlDOUL9bl6s+XERS9G2Y0ng== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708996754; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gZuSEC0XkXFwcYNwknnpwoG7sl6iWmfkB7gqgeAgx28=; b=Jx4c8ObXXsbzB6wvPTr8LFeKLB/stojbt5eybqVTN6xRN+i4qrQP3onoX8WpEWZ95FPRHC qIMeqq+lNGcZ6hICPB+DcFHazQUp7h5rpgOoESUjJfFGCCMyfUj5CUX/zBu3yEjuEjJx+4 pPvu/AD9R1DfD9wRYjt/qVEaulMRTdH6ld1V3Aip8gUJCQAuXSc2P6G01UkF4pqsY0LG/8 mBId45QkLD/F8I6Py+EggwY6XRxyMUvuyZdqkqHHPzjd6E8jHfTBKiCRvE7do9US1+FTVo tcV8QK5/pm/zzcbAx/y/Zs+2iIi6+95gYSFsqYRMLau0hnYXgwi6gly6CWsl+w== Received: from [IPV6:fd:1965::32] (unknown [IPv6:2600:1700:ab1b:6800:2e0:edff:fece:8f27]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: leres) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TkKQ95MXRz1MNy; Tue, 27 Feb 2024 01:19:13 +0000 (UTC) (envelope-from leres@freebsd.org) Message-ID: <9a5ad043-5183-4b3f-858a-1b26fb27b9aa@freebsd.org> Date: Mon, 26 Feb 2024 17:19:13 -0800 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: net/libpfctl: needs to be updated for 13.3 Content-Language: en-US To: "Edward Sanford Sutton, III" , stable@freebsd.org References: <80a7ec56-0e9d-4f26-bbe1-427c8672045c@freebsd.org> From: Craig Leres In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 2/26/2024 5:07 PM, Edward Sanford Sutton, III wrote: >> While building a bunch of ports under 13.3-RC1 I find that libpfctl does >> not (build). I freely admit I do not understand how this port works; I >> started to create a PR but I don't have the first idea on how the >> distfile is generated. > > `make makesum` or hand crafting it. If you update the port's Makefile to > be a different version or use different/more distfile(s), you can tell > it to update that file with this step. Sure but MASTER_SITES is set to LOCAL/kp and it looks like he hasn't generated a distfile for 13.3 yet. (Maybe this doesn't happen until 13.3 is released?) Craig From nobody Tue Feb 27 01:50:04 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkL5x3yLyz5C8fF for ; Tue, 27 Feb 2024 01:50:13 +0000 (UTC) (envelope-from mirror176@hotmail.com) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12olkn2067.outbound.protection.outlook.com [40.92.22.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkL5x16vHz4sZ4; Tue, 27 Feb 2024 01:50:13 +0000 (UTC) (envelope-from mirror176@hotmail.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gINmg0zYE1PpZqEI7w//h0jIl72GfQryKxoO02N8gNtzmm1lWEdtWk8ok4DeU1dd2/0SZorRY1S8049U5qoRgzUVAt1Ww859Nn79AQGfgKmC8jpaj0+2ZfUbm3Rh0xrdYkl0Ot9J4sdW7e9LCwD8ov5HEDcw7euaui3OLNev/wBA+NWHe4VCK36mI+w8yBJ2tsgPK1VyKxFT7uVTodYacHboKY7bNQmwZWuyS+IMkYnsw1m7XTR3h5L8zeP1mKCnyr3OWC5Ri4BZjn6wLjRgJZhI99OCunXoqXTKoJGZsjap3n0SNpgNzqMXHtphSF4qpoSSDOrnxuwuuoAVTKKt9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qIXWZ8CVL+QJ0Ty7dNZd+jXE/COiRy3bmxr2GCjTD78=; b=nU/0/FZYY7AKRUtiJc8hRJyBYv2XgUDXjxlv9BNEipgUJii/cZMhMqSth0TPzh2qCtZP3S5F1jw6us2Q7nXahM45waSml90GNgYsls7H4HtsOLKo5J4if/Uj9YaszHd+zEr+YK1LDdf29xdwIcjh1zIfkL4viN2E9yw8UdxsI+7L/hW65IwGm+BP6Sc5jIzgyziS+gG/vw+b+zxl95kUdPluOBf79GN+dzKIHwqLfRT49WjmTAkPE1OwoA5jaur8s3aWLzVc4/rqNTinok621WE+dMftlAcvZ3bWzzLh3bYR798ACI06ZydaxtSpxfJUf5Yhr4XKOngKXnd5/28ahQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qIXWZ8CVL+QJ0Ty7dNZd+jXE/COiRy3bmxr2GCjTD78=; b=a1rpkLUF6Pu31v0TBrJn3rzS3r9L45pI4IYOgNjWDKIwFdYLCdWZgqqJ/p67PpTCD7SlSaK4X18iGia16WKceIlcGYdSxttMleR2fn38AKi2pQcB2qukWZVDRb9qVvdDf/2ub1tYxqbCXAnEKjQ73dgIOTW0sxOS0NUN3ADZfvJY1fid1kZ3Jw7CJZsjv6H0yJ45oKmhhlYJ7Pr3gU98h6Y0pZNxls1dAv4WaiOhKQp/TBmMMhD22xJSBzpWeKhMYtTqG4rZycqs2OJ4AJ5XuyaRcTYVksXd1SiHKMuioxFtoqiYvd48UCqMC82Z/6DhTNW5o7ccY0AxRb8GZJiL4Q== Received: from CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) by PH7PR11MB6858.namprd11.prod.outlook.com (2603:10b6:510:1ee::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.22; Tue, 27 Feb 2024 01:50:10 +0000 Received: from CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::e526:b74c:4798:1295]) by CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::e526:b74c:4798:1295%4]) with mapi id 15.20.7339.024; Tue, 27 Feb 2024 01:50:10 +0000 Message-ID: Date: Mon, 26 Feb 2024 18:50:04 -0700 User-Agent: Mozilla Thunderbird Subject: Re: net/libpfctl: needs to be updated for 13.3 Content-Language: en-US To: Craig Leres , stable@freebsd.org References: <80a7ec56-0e9d-4f26-bbe1-427c8672045c@freebsd.org> <9a5ad043-5183-4b3f-858a-1b26fb27b9aa@freebsd.org> From: "Edward Sanford Sutton, III" In-Reply-To: <9a5ad043-5183-4b3f-858a-1b26fb27b9aa@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TMN: [9JNGUqZLIV4pwzFKE3G0ekciOQTwxw29] X-ClientProxiedBy: BL1PR13CA0098.namprd13.prod.outlook.com (2603:10b6:208:2b9::13) To CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) X-Microsoft-Original-Message-ID: <92a376fc-3916-461d-8bc0-29017eea9a92@hotmail.com> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PR11MB4770:EE_|PH7PR11MB6858:EE_ X-MS-Office365-Filtering-Correlation-Id: c5acfdc8-2b86-493c-e95e-08dc37366e6e X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AeAmhiiYWINGcdA9VEupeZP56J1ZlgDtbzxZD3RuK9LTcW9/1TqbKfmfSpJCtneduKKZPbKEdR2mDd99Xyl6DkJoV4VLJKx0mRta7KU0nf/zDtGuw+YFQBJafSXH4DCOdtz674wL7INc9484XimVaI0B5gzsgw7AwnJfo4jLmLRXlGHApWIcOthB4+8xVcORQ8vT9XtiEF0ZStx5Jb/23VigFCTgRcoYReAxPVdNyntzyExVF0rJVO+EzQhMXN5zdz2ShjFmptF+ryOAEFEgutNS+TUog4YEGpJ3Kn8E/R3COWXYE7pJxBkSgAcpge+LF8euS+XWLC77aW0aO1KdqRMDgkMs4m7Wzv3DnnOz/qi5C6DUXBdJVinbVS9USz+K3RleL0/G97OmkoRtJ2MPy7jNnEqwcXxyzzNEvxJd28BaUuj5O/lRx8AsmrGx8p136QAAYHddWvK3wIVmhKc+UAyS/cPCvjwQLDzVUoaWqGbpX+1EcxGhkE53W4NQsZSzNUxWZ/cbrFyDTodrTw+mKsbY39klZXyd3IhRAs+oQoo2uyp/OYyeSq4hSJCfnpCx X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YTROQzdSV3Q1cnJ3ZUZ3Z05nSzhSYXBvRHIyTWQ1WDROaERuMmsyeGJVdGUx?= =?utf-8?B?RmJjUjFycWNIcFZvZVV0UCtNbG94dmlDSWMzdFBuckNTVWtDbW02UENrNFBX?= =?utf-8?B?d0x5WFl6M0lkOE5pczkzcTJOSjJFMElCTzQ5aDRVeXZyYlNUVFp4VGpJWC80?= =?utf-8?B?L2xLQ25qbHAxdHpLVGxiSHVKVDZ0cUU4TEpHSG4xRHpoS2dnYWtTOFluaEVF?= =?utf-8?B?bVB3Y1ZYTCtyK212SG4ybHZzRHhUTmtTY2Job0V0UnJNRkdZTk84THIrbU5G?= =?utf-8?B?R1lwWlhaclFMVWlsMm5tOGpRa0JpdHhReXVzTFByZHpOU09vUGQwSitiQXk3?= =?utf-8?B?dlk5T2dCTFZTWERMNWFpYlIwZ2pUNGE5SzhxbTBIWHdHdXl2T20wMGJLTDc0?= =?utf-8?B?Z3BmMzFEMXIwVVZpTS9hbnNZcy8vWDVUUmNIVUhFUE9IM2dTcjAxYVRsT3pw?= =?utf-8?B?TXMyallCNk1DdWFraUEvWkMwU0RTVEZLeWM0ZXdtT2JkSk44eStiK1NwRWR1?= =?utf-8?B?ZCtyUFlXWWpBaVEwWWtmMFM5WjdnTjJMdW9ZMVZ5N3ArVGpUU3FLbFgvZFZt?= =?utf-8?B?SUVLalBjclJ4eEtlSTFpR05tQjJDRUptYUR3emtDWjBWSnNVamtjdXRQb2xL?= =?utf-8?B?Znp0bnkyWk96b2hGV3ZYLzAwZTBmTWljdXBqcUdybCtWVGZaMjVrZTRlcFRT?= =?utf-8?B?T1FNL2E5ZWluM0MrRDhqVVVBanpNeFVTMVpTMnBaWCs0Sy9vWVNtOTZGZXdo?= =?utf-8?B?cEpzWFZqd3A3WTJDak1OZjcweWhxTjM3eTRWNW5DZlAzMkhVcVZlNWFyQjFr?= =?utf-8?B?bjJqSzVWc2ZRQUlPclVKT2Y3enpPcWVKY1FTY01IM2QxWVROL1U0ZnFIcW05?= =?utf-8?B?MGhlNjRCRnhxd3YvMmdnQ3kvVWN5UWlFaHU3MzBXWmZQSE9lb081RTcyV2hO?= =?utf-8?B?cDNDanJhdFU5V0h5QVh1UjI4bTgrN3F3ZTNrWHBNbzB3MUlIem8xU2xVem1Q?= =?utf-8?B?VVRwVHJSdjV1bWdMVE1zWW5vY0I0TTltNm0xTGUxQyttZ0pWWnIrZ3FHT0NH?= =?utf-8?B?cGlqK1ErVHlTNm1GWTAzalFCUDFOZGJTL0xaengzV2E2TFcrayszYkVxaTBM?= =?utf-8?B?eklQNERaMlhOcWgvcW4xNDdkWnE4U2NVWUZrZFBBaTJCM1dPT1R4M0FLY0lJ?= =?utf-8?B?VTcxZzAwdHdKRlgrcnNCaUJ1SXVMZGcvS0hwS1AreUhrWjFsbEhXQmtrSnZv?= =?utf-8?B?WnBKd2NWa0NoSFQyR25UZE9tcWxSb0VQSXhxM2JrWEl2NWE5S0Fra3hFT0Ja?= =?utf-8?B?d3VkQ0RYUkNielUzRXhLWlZ2NVQvaW4yQWhwQnNyMUtjQVlMbVVnRjhLQkdF?= =?utf-8?B?S3hsRjNSNmJsc0pKd2hMRUlHdkNoTWJWTlZ4bXBqcHMwRmN4N0gzTExGZzJ5?= =?utf-8?B?cXFhVTU1Ujc4L05JaW5nSkpCWXlGUWROYzJLUmVqUnNOTmZqWEY0NHZwZEln?= =?utf-8?B?Q3B4Vk02eGdMbmZaU1pGRWVOMnlVb1B3WlBlVmpLNFBRVzBldkdrVERXTnUv?= =?utf-8?B?V3EvMisrN1RpOXBwQkVOamYzbkpad3R2ODZENUxWekttVGlVMC9CTklwbnhp?= =?utf-8?Q?FzwhyUnNtrSqpZZ0LQuYfL2UopitCUjG709MVMWvfmH0=3D?= X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-e8f36.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: c5acfdc8-2b86-493c-e95e-08dc37366e6e X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4770.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2024 01:50:10.2923 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6858 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US] X-Rspamd-Queue-Id: 4TkL5x16vHz4sZ4 On 2/26/24 18:19, Craig Leres wrote: > On 2/26/2024 5:07 PM, Edward Sanford Sutton, III wrote: >>> While building a bunch of ports under 13.3-RC1 I find that libpfctl does >>> not (build). I freely admit I do not understand how this port works; >>> I started to create a PR but I don't have the first idea on how the >>> distfile is generated. >> >> `make makesum` or hand crafting it. If you update the port's Makefile >> to be a different version or use different/more distfile(s), you can >> tell it to update that file with this step. > > Sure but MASTER_SITES is set to LOCAL/kp and it looks like he hasn't > generated a distfile for 13.3 yet. (Maybe this doesn't happen until 13.3 > is released?) > >         Craig I misunderstood thinking you were attempting to update the port and I had not looked into it myself to see that it isn't just using some obvious donloadable file. If that file doesn't exist for download yet, then there are other steps necessary for us to have a file to point a port at. I had seen that fetch failure in my builds but not started to diagnose it. As I do not have read permission for that folder on the FreeBSD server, I'd defer that to the maintainer to sort out. Looking at the Makefile, it looks like files are created+uploaded in the 'upload:' area; maybe that could be modified to create the archive for 13.3 and copy its output to a local distfiles repository to test; if successful then I presume the maintainer or someone with write permissions to that part of the FreeBSD servers needs to execute such steps. `make makesum` would erase nonreferenced files if I recall, so I presume an update would need to append or edit the file to contain information for all the versioned distfiles so that each OS version grabs its own needed file while ignoring the other files. From nobody Tue Feb 27 01:59:59 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkLKg0TL6z5C9lY for ; Tue, 27 Feb 2024 02:00:23 +0000 (UTC) (envelope-from vester.thacker@fastmail.fm) Received: from fhigh7-smtp.messagingengine.com (fhigh7-smtp.messagingengine.com [103.168.172.158]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkLKf41lVz4vCF for ; Tue, 27 Feb 2024 02:00:22 +0000 (UTC) (envelope-from vester.thacker@fastmail.fm) Authentication-Results: mx1.freebsd.org; none Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 00A5C11400F5; Mon, 26 Feb 2024 21:00:21 -0500 (EST) Received: from imap51 ([10.202.2.101]) by compute2.internal (MEProxy); Mon, 26 Feb 2024 21:00:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.fm; h= cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1708999220; x=1709085620; bh=JXo6fZM01x Ul3udQadxmQnc08VO3xgKKsM0yudaFct0=; b=JaNwlWKMYuMi0NxPPwQBWvnoze H4XlvhEQy/sBjmXYs4YC2EquYv52iu3/8A2quBEgTRM9iqTWNEy6ZcLfrLLFlGw+ sgvT4qUF80e2eMd5steeh/Jj3Ft08iwWd51ZqtrIimG4jnheY6vTX8u6PijnFtgJ GZoDoOgpO2Gp35g48I1wGodOStXzJMmMPQUTMXzOjnry+bZQF7IOoga+3NfAYtxa vmhMQG9GufRqX6vd22llye9et5DZN4tob+udGa7b6K6DJZVNWmGSH2INFbob8GE6 SRAGxDBh6S7N50xZeMPgkYOuZL+V6f5JEUJpUSRHsBxVKsDohmuXg64p2kNQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1708999220; x=1709085620; bh=JXo6fZM01xUl3udQadxmQnc08VO3 xgKKsM0yudaFct0=; b=kylBYN1M8fgok5KXOh4/+K4XtaMsv1s2hvGFPyIFaTSr Y4Qtp8npoqrxKFMTfDAfas/jZ3MZgc4G7ZhmSL4+iYz+6dbj92cdGsdEAz2tRA7l 4gfDLWTel/aJrNa2KL75oHzjbyyxy2b6B8YZE+sGbivWZ+59LcDi9EjjbZCaPnEQ m1uCNRYjgopWzzBIsp5q/ApSfmJKwjIfv1PBMv04ovWC3hOSyG/6T60msf2q8IDi mfC28wiRhDzR+tuWeQOW7CYyLyi8xz6Sl4sFXhmkG1+f9yo3WdC5SeqXdW4PSQlG YDgb6Akda5pfWS9RPaOWV7BQiLOXeOMj/XZbr4n9mA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgeefgdegudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefofgggkfgjfhffhffvufgtsehttdertderredtnecuhfhrohhmpehvvghsthgv rhdrthhhrggtkhgvrhesfhgrshhtmhgrihhlrdhfmhenucggtffrrghtthgvrhhnpeefie dvudelffdtudelgfffiefftefghfejtdeiteeiheffffetfeegjedujeeiffenucffohhm rghinhepfhhrvggvsghsugdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpehvvghsthgvrhdrthhhrggtkhgvrhesfhgrshhtmhgrihhl rdhfmh X-ME-Proxy: Feedback-ID: i219c416b:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id CEC09B6008F; Mon, 26 Feb 2024 21:00:20 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-153-g7e3bb84806-fm-20240215.007-g7e3bb848 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Message-Id: <65607177-0f4f-43fb-acef-ecd89965717b@app.fastmail.com> In-Reply-To: References: Date: Tue, 27 Feb 2024 10:59:59 +0900 From: vester.thacker@fastmail.fm To: "Edward Sanford Sutton, III" , "Robert Blayzor via freebsd-stable" Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task Content-Type: text/plain X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US] X-Rspamd-Queue-Id: 4TkLKf41lVz4vCF Given the complexity of your situation, a combination of adjustments might be necessary to find a balance that works for your specific needs. It's often a process of trial and error, tweaking one setting at a time and observing the impact on your system's performance and responsiveness. The issues you're experiencing with system responsiveness while running port builds with Poudriere on FreeBSD, especially with high CPU load conditions and specific system configurations, can be complex and may involve multiple factors. Here are several areas to consider and investigate further: 1. System Configuration and Resource Allocation: - With your system having an i7-3820 processor but only 2 enabled cores plus hyperthreading, there's a notable limitation on how much concurrent workload your CPU can handle effectively. While hyperthreading can improve performance for some workloads, it might not be as effective for highly parallel tasks like those encountered during port builds. - Consider enabling more cores in BIOS if possible to distribute the workload more effectively across the CPU. 2. Usage of `nice` and `idprio`: - Using `nice` and `idprio` to adjust the priority of the build processes is a good strategy to maintain system responsiveness. However, if the system is already under heavy load, even lower-priority processes can contribute to responsiveness issues, especially if they are I/O bound or involve significant memory usage. - Experiment with different priority levels and observe the system's behavior. Also, consider limiting the number of concurrent jobs (`-J` option in Poudriere) to a number that's more manageable for your system's capabilities. 3. Monitoring Tools: - Utilize tools like `top`, `vmstat`, `iostat`, and `zfs stats` to monitor system performance in real-time. Pay attention to I/O wait times, memory usage, and CPU idle times. These metrics can help identify bottlenecks in your system. - For detailed ZFS performance analysis, consider using `DTrace` scripts that can provide insights into how ZFS operations are performing under load. 4. Software and Security Updates: - Ensure that your system is up-to-date with the latest patches and updates. While the advisory you mentioned (FreeBSD-EN-23:18.openzfs) may not apply, staying current can help avoid known issues and may include performance improvements or bug fixes that could indirectly address your concerns. 5. Hardware Considerations: - Although you're already using an SSD, ensuring it has enough free space and is not near its write endurance limit is essential. SSD performance can degrade as they fill up or age, impacting overall system performance. If using ZFS, here are other considerations: 1. High Kernel Activity with `arc_prune`: - The `arc_prune` process is related to the ZFS Adaptive Replacement Cache (ARC). High CPU usage by `arc_prune` can indicate that the system is actively working to reclaim memory from the ARC, possibly due to memory pressure or intensive disk I/O operations. Since you are using ZFS with possibly high compression (zstd at best compression), the workload on ARC and disk I/O could be significant, especially during intensive operations like port builds. - Investigating ZFS and ARC settings may provide some insights. Consider adjusting ARC limits (`vfs.zfs.arc_max` and `vfs.zfs.arc_min`) to optimize memory usage. However, be cautious with adjustments to ensure they fit your workload and system capabilities. 2. OpenZFS from Ports: - Trying OpenZFS from ports might offer newer features or performance improvements, but it's crucial to ensure compatibility with your FreeBSD version and to have a reliable backup before making significant changes. Changes in ZFS versions can affect performance and behavior, so this could be a double-edged sword. -vester On Tue, Feb 27, 2024, at 09:24, Edward Sanford Sutton, III wrote: > Currently trying to do port builds with > poudriere-devel-3.4.99.20240122 on 13.3-STABLE FreeBSD 13.3-STABLE > stable/13-n257396-134580c103b4 GENERIC amd64 and have had poor system > responsiveness on this and a -STABLE that was likely at least 2 months > before it with `/usr/bin/nice -n 18 /usr/sbin/idprio 31 poudriere bulk > -J2:12 -j local -p local -f /root/installed-port-list -f > /root/prime-origins` and also launching it with no nice change and just > starting with 'idprio 31' (in case it would react different with the > builtin idprio. It used to be that under just idprio 31 I could continue > to use the machine during builds with minor impact on responsiveness > with MAKE_JOBS_NUMBER=4 on a 2 core + hyperthreading i7 which lead to an > oversaturation of 8 build processes across the 4 virtual cores yet > stayed mostly smooth. > Responsiveness seems to have strange effects of laggy mouse/keyboard > input and thought I recall logs saying system couldn't keep up with the > mouse communication, programs intermittently freezing/unfreezing, and > even building becomes unstable with electron28 failing as runaway > process after over an hour when killed as runaway process during extract > phase. During the lag, programs seem to take many times as long to > respond while also getting high cpu use during that time such as tmux > switching between windows; ctrl b+ctrl+p getting almost 100% cpu core > time for about 5 seconds added to it under top. blacklistd has take > about 24 minutes of CPU time for < 3 days uptime per top while managing > a usual slow bot break-in attempt on ssh. > More recently looked and see top showing threads+system processes > shows I have one core getting 100% cpu for kernel{arc_prune} which has > 21.2 hours over a 2 hour 23 minute uptime. Previously I know I have seen > higher system % time than I'd expected but not always sure when it is > justified or not. I started looking to see if > https://www.freebsd.org/security/advisories/FreeBSD-EN-23:18.openzfs.asc > was available as a fix for 13 but it is not (and doesn't quite sound > like it was supposed to apply to this issue). Would a kernel thread time > at 100% cpu for only 1 core explain the system becoming unusually > unresponsive? cron was stopped after last boot so shouldn't be throwing > up any unexpected background work. > System has 32GB ddr3 RAM with i7-3820 processor with only 2 enabled > cores + hyperthreading in BIOS. Issue appears specifically when CPU load > is high and idle% in top is 0.0% but it is only sometimes present under > that condition. I usually use ccache and WTIH_META_MODE to try to speed > up compiling base and ports and have zstd at best compression for > packages in hopes of faster extraction at the tradeoff of more disk space. > I haven't tried yet but considered trying OpenZFS from ports. Any > suggestions of what else to look at or watch for? > Thanks, > Edward Sanford Sutton, III From nobody Tue Feb 27 05:48:29 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkRPF0jWNz5CW8V for ; Tue, 27 Feb 2024 05:48:49 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-22.consmr.mail.gq1.yahoo.com (sonic317-22.consmr.mail.gq1.yahoo.com [98.137.66.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkRPC3KFrz4M4C for ; Tue, 27 Feb 2024 05:48:47 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=s+cGj1zR; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709012923; bh=uVXGLF5N0AHKv2im6hT5MdheqBTmjuYCWpQGu958xtw=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=s+cGj1zRIKcaxmo/PsJGB/jnYlzdKxzTOkxU4cPvhIKv7yyL4zv1oa5JDHxUfqO9pf63dUmy8IHGMlwTEE/LVjQwXGGmm1ExHNhquQuGtd5NzL5sZuUvJ+oePaI1AImqaVICl3idfwZrlG3+GiRUZtPwAZe2mnxTxyLk83emMmbHW69NfboMjafPnI/GUJ53788MclKY2HwbyEuUurOZM17WYJfKcmiA05L46X+B1kiJneD64X3bF0b5VR94qcnO7uHamTXacXt+jmQUwGNb5rXozYLaxsJ9JZKN2V8WRKM4RnHNmOFTixbT9xBP/YO3IuCKeDZzAoZCOO4QcSBDYg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709012923; bh=39oaBlRT/3DxohMOj+xeURiKkt4341PuNz6yFZbYpOa=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=kGDI7S0U0MbsUKIXcC4eC1PL00HAX6NOEaZGIMpgGyrmvXUkkyDm/U3gpaCYfI6GSRI98mflBnwWcxzyJGaJ+y4lfZ5zTA6Nfhcn167tZgtWT456SD/ppKWpLaOD2wDcz5S4hnjf4kBPir1swWxCxjCqhxJlXNNHdE2TPil0OUsCOXIXFxddh+QjySqGbj85t5FR1nEZY4LX7qXpIVuPSphNNplSzgqRhfS8Mw8Vby/Tq9JjgLnZeh+6vlKJetFWtdSTxTZ5FBZqwV8C83wr6WWs/OCsMfGO4h59p8y5J78huMly9jo7FJ4WOAPTccPGe1fr38Nz8cl4I3AajIS7zw== X-YMail-OSG: kMR9xeIVM1mh8JlqG8yKEx1x39i5WBbQcJIRlfytpoqphbrGNPKUiE4kpItFhyq g35j4Hpj..p2dfcr9Bop9PlVT_xJJZIQEpGjA7XBolUXu6YewJ.UFERWlxzldsYDTryjHYyeASVx EXGpbF76m..I63SaPrC_34iGdr7aW45F0Y0pwVheS01PuUlT9p3jiucGPnMUPY1mbvIWxA7cD2VH n7Kt3Cj5EQl0UpeBUAX4CAd5GBXkS4d2NLV_MVaQz6.qQMaG0lcD_zQCshYdYJpAvav0Al3aqO2P 6gSGYCold2TvvLdnmAYcxRyoX_Z6IDkb8KcRgicXIzmNxvmhMp7RzrsAwLEZAbFEIkS8LUAJlzdp xAfuggAWyW5bcJwdHTLC4sqBA13uV4nB92eyFuPgJ0nrj_NTk1mtVxbw1V_XB7KiQ5JmqlyK8hEq BpbzxLl3XiFRMEijHIq3UWmS0zgB2T5BzCDxX69PiY0mBLFiAHhUolZNAirlAanBE0DQXEAqolY1 noyfGO4wz89u155voLdLwLBqSyxXnEtr3nnTpICdhiTA088GAJsuLTxnCsD2LUYiGjaPB2pGOG6I vQVtRjE7NRpHR09gtqicf8oBLlQL7Pw1HgmBINaneKYmmU6K8w7a7lAprQgfITkzQOymIJRZC.aN e6ow2OPzCCEsjjcxu3kJg8M_hJKu6Nt0M8y3OA4TJVrZBim1DVx9wLhFtdlec58naTjDTy1aAyM7 9Dp1DhlygJGpOiC3xM_HHr6FYN8RC_TvJg3o83ka.KicWzcfApWUWeHOG945p1CDJytd8zi.MZ0J j0w5qla0oun6OPLP1QBDEaqKpwERj91Zcr1mGafJssnm7XtzS5TAVGuYBkUSJnB8yM4el0xaVi7Y NBtO9kPrPPdIglpdP0yoKz9RSjJTDRJlnBiqEB9_kEuQswp1z8k.HOTUDHeJjBMgYqoLFJ0PBNyv y6CcK5soVdlPmGHW.6.l3S_zOGpMJQ_VDUECCDZ1D.W52BuGbeMfNTKcNSx8tEFbEgh5X1OGzftw tfXSjxlBGFKm72cLMxNLY7FDE12lBHCGjqiAnN37jQSX7rZCzNcTtElPKBH2G57hR7kMWwtviV_G tmQ0xt1Qu6CyFcj2e5EgdZ0kvUef6UnvMDEokrJXw881uoGZna3QHP88NP3yO66r260AJgNm8qaE W5ys5wkwJOY.CETBBHLKGjUX3iV8luJBC777dhPqybyto_ZZME.jPn0fIi8u_54QwajrwhH65rNi 4I_tfXR5UAeIAgJSE4P9cJewXEdhZN1ZJ11XLcRaSAZStr5dlDmfEUQabhMbr2V7VbOIp_yQpPh9 .hkwYdm1C3_VR5ZTyHpN.K4CAg4Nqiv4uOFxFI.beK2gtKOcbzZmQudidJFjI.8OSHEGyQCZDM2N eX4XbcHPpfsKpN.kbPP5ES68deKLeUdSgkAtPinLh_Li.T8.aVwCFoEzJeCVQLRHK5id_dIxuWQd cxkVP6hlgYuX.n6pootE15PtqYwUKY4sDdvbI4iZLpeXSTaXGKm4pc0Oq_IZPECb0ELWEg057kYQ BLs._S9m.0rXzQh7wkd.mEpmmkt5VXkZH4KV_t3ZgteKxgmo2yEZvA5.7Ku2tTM14l_ieykHdOFY 9_2auyQN_t2mY27C7jiC5xbyHK_6swcHdB9_qq1m.5I01a5.87AiLopkrAOa_25xvXqzZark.UJg xe9HGhaKGNXgCYYa76E_ZlfVv08kj9k60QhkZQgSqOwcD6REz0eVLuE437yx8vuJhGp2VrDwA19J b7F34v1CiVZPSet51fjUdJR0LdwzRcKvmT07K1Lc9vGbHVnidEuaeoCMUQiY6EyuyjV5uP2miZII QDOAe0sUacjZdxt9eHADrANdJNKe3_fcWpC4gTYfBrBr6STF9QOd.VmNTQo._klG_PD7xC_51lbT k08jXVuMVEQemie9YVUqT.5GAxdJTATjMaHwbKurofV88M2PF76A910zPCLLJx73j8fVbW3Fa4F1 kM8zC.Nh7Nyf3PZopD7Ec5LMjvWweOFd7Ia3jB8CceuUnAJJex40CuPDAeaHb2QOkkY9S6U3xqAb x6tHoqEuJOIKn0lJvIbbz.QQEB_36x4v154avFjRowl2hSOOsbJirnUOJXvjOW06sGY0j.6l1zy9 NilvOI65a8XZvIgFTj50Tb36G019.E8ULL9AK59F7wkOBbX4gzneWB.X0j7op5N11xerD3GbrxgJ mBvPQHbjebMjK__TvN6P2Quty2HkZaJS4LyzeB2VXdJ8NyotDX_Ws9Xu2gFlcL13y2HWr0N5j3bs CvOb3 X-Sonic-MF: X-Sonic-ID: 6a10d858-4d85-4e7d-a17b-2982201dc2dc Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Tue, 27 Feb 2024 05:48:43 +0000 Received: by hermes--production-gq1-5c57879fdf-hjdnf (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 28662a6d25ef6001c0d8fd5d4466a3ef; Tue, 27 Feb 2024 05:48:40 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task Message-Id: <4D5E09B1-EA6A-4966-838D-0D003F997953@yahoo.com> Date: Mon, 26 Feb 2024 21:48:29 -0800 To: "Edward Sanford Sutton, III" , FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3774.400.31) References: <4D5E09B1-EA6A-4966-838D-0D003F997953.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; SUBJECT_ENDS_SPACES(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; FREEMAIL_TO(0.00)[hotmail.com,freebsd.org]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.148:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.148:from] X-Rspamd-Queue-Id: 4TkRPC3KFrz4M4C Questions include (generic list for reference, even if some has been specified): For /boot/loader.conf (for example) : What value of sysctl vm.pageout_oom_seq is in use? This indirectly adjusts the delay before sustained low free RAM leads to killing processes. Default 12 but 120 is what I use across a wide variety of systems. More is possible. For /etc/sysctl.conf : What values of sysctl vm.swap_enabled and sysctl vm.swap_idle_enabled are in use? (They work as a pair.) Together they can avoid kernel stacks beings swapped out. (Processes still can page out inactive pages, but not their kernel stacks.) Processes withe their kernel stacks swapped out to storage media do not run until the kernel stacks are swapped back in. Avoiding such for kernel stacks of processes involved in interacting with the system can be important ot maintaining control. This is a big hammer that is not limited to such processes. Both being 0 is what leads to kernel stacks not being swapped out. For /usr/local/etc/poudriere.conf : What values of the following are in use? NO_ZFS USE_TMPFS PARALLEL_JOBS ALLOW_MAKE_JOBS MAX_EXECUTION_TIME NOHANG_TIME MAX_EXECUTION_TIME_EXTRACT MAX_EXECUTION_TIME_INSTALL MAX_EXECUTION_TIME_PACKAGE MAX_EXECUTION_TIME_DEINSTALL (Some, of course, may still have the default value so the default value would be the answer in such cases.) Also: Other system tmpfs use outside poudriere? ZFS in use in system even if poudriere has NO_ZFS set? (Such is likely uncommon but is possible.) (Other contexts than poudriere could have some analogous questions.) For /usr/local/etc/poudriere.d/make.conf (for example) : What value of the likes of MAKE_JOBS_NUMBER is in use. Note: PARALLEL_JOBS, ALLOW_MAKE_JOBS, and the likes of MAKE_JOBS_NUMBER has as context the number of hardware threads in the context. The 3 load averages (over different time frames) vs. the hardware threads for the system is relevant information. Note: with various examples of package builds that use 25+ GiBytes of temporary file space, USE_TMPFS can be highly relevant, as is the RAM space, SWAP space, and the resultant RAM+SWAP space. But just the file I/O can be relevant, even if there is no tmpfs use. There are questions like: Spinning rust media usage? (An over-specific but suggestive reference form the more general subject area.) Serial console shows a responsiveness problem? Simple ssh session over local EtherNet? Only if there is a GUI present, even it is not being actively used? Only GUI interactions show a responsiveness problem? Going in another direction . . . I'm no ZFS tuning expert but I had performance problems that I described on the lists and the person that had increased vfs.zfs.per_txg_dirty_frees_percent had me try setting it back to vfs.zfs.per_txg_dirty_frees_percent=5 . In my context, the change was very helpful --but, to me, it was pure magic. My point is more that you may need judgments from someone with appropriate internal ZFS knowledge if you are to explore tuning ZFS. I've no evidence that the specific setting would be helpful. There has been a effort to deal with arc_prune problems/overhead. See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275594 === Mark Millard marklmi at yahoo.com From nobody Tue Feb 27 16:05:03 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tkj4m0SFCz5BxjQ for ; Tue, 27 Feb 2024 16:05:28 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wfhigh3-smtp.messagingengine.com (wfhigh3-smtp.messagingengine.com [64.147.123.154]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tkj4l1HsCz4b6s for ; Tue, 27 Feb 2024 16:05:27 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm1 header.b="FwAXq/4N"; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=GJzn8VhO; dmarc=pass (policy=none) header.from=f-m.fm; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.154 as permitted sender) smtp.mailfrom=void@f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.west.internal (Postfix) with ESMTP id 3E9131800090 for ; Tue, 27 Feb 2024 11:05:25 -0500 (EST) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Tue, 27 Feb 2024 11:05:25 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1709049924; x=1709136324; bh=BIIqrwXuQN A7WCCgktU9h0osYsFMTQNCW7pU+xsrKxM=; b=FwAXq/4NZGslV2hagZn7W+jQuW s2LhmfRLLOIQfsaZA8qnp330NxE2LqcKZZpkAm2Z6ml6lLPgialYRojTT4U6zCZg f5gTXOmpNu4Oxmb5/GXx/57LNXEuiohiuLUgJyMXBPOmbvebF3Ib4QIj5RU817ty 7f36F2FFDTWY+GbjIF8TIeqHiszE8hFaJVX/HIqlQSrLDXzIiAR5cIpzLldEOoxW Ynw4aGWgPqz4nYQPMuzQBZ2bOb+egUBsDGnWs6qHa1DBKfh1x2m+1i93cErrfpZY jvQS1HBUgELDgequ2ePgX/R9s53/xuuupviX5T9+WBv9Q3pPT1aYmhFB3sVQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1709049924; x=1709136324; bh=BIIqrwXuQNA7WCCgktU9h0osYsFM TQNCW7pU+xsrKxM=; b=GJzn8VhOysoWxB+9S3yPlRuvJ2S/umdkIvXaJLpIglXe XuQTswXtiDbCiwURDmiYi9jdy+NXMUgLCteI/H5sXEYDZRUMJF/TESFaOJ43VCvx b+56pjw3EMLndnB9F6aaBDjN8MPiK+mX0RRuVXJMJ/bDPpuhgoI6em+hKYHrrXe1 VYfyv0GZ1EpHOMGw/46sPwmK+2sYQ05HL6MM/boV2dXDZT/kud9dgqZdgwQ+FanU 9dwCBJRBybgptpdEUhBSvXbYqQLvZcvy5WBhU41bqAtf47lCy4EXMVWhEdeGosbI X+pZmlymh5vwNbY2QIeYhfG/GIAJHQTKnhlxUnssgA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrgeehgdeitdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfggfkjghffffhvffutgesthdtre dtreertdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepieetvdeuhedthedtvdfhuefhveehvdeiledvieffheevleehgeefudelje dukedvnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id EA8582A20090; Tue, 27 Feb 2024 11:05:23 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-153-g7e3bb84806-fm-20240215.007-g7e3bb848 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Message-Id: <6bf693a3-6fba-4bf3-a971-c71dbffd57b4@app.fastmail.com> In-Reply-To: <4874751708942864@omv3tj5zwvd3plqc.iva.yp-c.yandex.net> References: <5813111708783942@wckjnuw7tehz7zgt.sas.yp-c.yandex.net> <3c8667a2-770c-4c35-9b33-572f43d9f1ea@app.fastmail.com> <4609701708855679@vpfns56j3qv6ec2k.vla.yp-c.yandex.net> <511954d9-74b4-45c3-b16e-342ad29751f7@app.fastmail.com> <478521708877670@z7oys7c4urhdnodv.sas.yp-c.yandex.net> <5a581a59-94fc-4476-97c4-4afac65c27bf@app.fastmail.com> <4874751708942864@omv3tj5zwvd3plqc.iva.yp-c.yandex.net> Date: Tue, 27 Feb 2024 16:05:03 +0000 From: void To: stable@freebsd.org Subject: Re: USB CD drive does not work with 14-Stable Content-Type: text/plain X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.56 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; NEURAL_HAM_SHORT(-0.47)[-0.468]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm1,messagingengine.com:s=fm1]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.154:from]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; FREEMAIL_FROM(0.00)[f-m.fm]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+] X-Rspamd-Queue-Id: 4Tkj4l1HsCz4b6s I get the following from my own portable BD machine, from the lsusb command: Bus /dev/usb Device /dev/ugen0.10: ID 0e8d:1956 MediaTek Inc. Samsung SE-506 Portable BluRay Disc Writer There's also lots of "can't get device qualifier: Input/output error" with -v but it's not stopping the disk from working. I can run k3b and use it. > can't get debug descriptor: Input/output error. > Does it mean that my USB hardware is broken? I'm not sure - it might be worthwhile subscribing & posting to freebsd-usb@ -- From nobody Tue Feb 27 18:26:06 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkmCB03bxz5CBg4; Tue, 27 Feb 2024 18:26:14 +0000 (UTC) (envelope-from serguey-grigoriev@yandex.ru) Received: from forward500b.mail.yandex.net (forward500b.mail.yandex.net [IPv6:2a02:6b8:c02:900:1:45:d181:d500]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkmC74f1Nz4v4D; Tue, 27 Feb 2024 18:26:11 +0000 (UTC) (envelope-from serguey-grigoriev@yandex.ru) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yandex.ru header.s=mail header.b=nnP0miBU; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (mx1.freebsd.org: domain of serguey-grigoriev@yandex.ru designates 2a02:6b8:c02:900:1:45:d181:d500 as permitted sender) smtp.mailfrom=serguey-grigoriev@yandex.ru Received: from mail-nwsmtp-mxback-production-main-32.iva.yp-c.yandex.net (mail-nwsmtp-mxback-production-main-32.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:9497:0:640:2979:0]) by forward500b.mail.yandex.net (Yandex) with ESMTPS id 83337610FB; Tue, 27 Feb 2024 21:26:06 +0300 (MSK) Received: from mail.yandex.ru (2a02:6b8:c0c:bc83:0:640:2077:0 [2a02:6b8:c0c:bc83:0:640:2077:0]) by mail-nwsmtp-mxback-production-main-32.iva.yp-c.yandex.net (mxback/Yandex) with HTTP id vPdOF0AoLqM0-nHtBvhI3; Tue, 27 Feb 2024 21:26:06 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1709058366; bh=t96i187CztBDAY8R9tvjKRo9ssffl0cESGd0z57AcK0=; h=Message-Id:References:Date:Cc:Subject:In-Reply-To:To:From; b=nnP0miBUJ2ZnugZA+gI7Bmax8mdWmrANR1I3AW4+AmYqIFasKLnrAthsK9TpY0YHO tDr8Hi9d+MqCMJE7fYcQyhXEr7lw1efxukcVJLp12S/khLy7H3gRQvuHsUcY9EPLpD 9EwdlacCyOQv1wsappC9I+GWDezkS+EBQIXY9CXQ= Received: by o5ump4w73ck7ppml.iva.yp-c.yandex.net with HTTP; Tue, 27 Feb 2024 21:26:06 +0300 From: S.N.Grigoriev To: FreeBSD Stable Cc: stable@freebsd.org In-Reply-To: <6bf693a3-6fba-4bf3-a971-c71dbffd57b4@app.fastmail.com> References: <5813111708783942@wckjnuw7tehz7zgt.sas.yp-c.yandex.net> <3c8667a2-770c-4c35-9b33-572f43d9f1ea@app.fastmail.com> <4609701708855679@vpfns56j3qv6ec2k.vla.yp-c.yandex.net> <511954d9-74b4-45c3-b16e-342ad29751f7@app.fastmail.com> <478521708877670@z7oys7c4urhdnodv.sas.yp-c.yandex.net> <5a581a59-94fc-4476-97c4-4afac65c27bf@app.fastmail.com> <4874751708942864@omv3tj5zwvd3plqc.iva.yp-c.yandex.net> <6bf693a3-6fba-4bf3-a971-c71dbffd57b4@app.fastmail.com> Subject: Re: USB CD drive does not work with 14-Stable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Tue, 27 Feb 2024 21:26:06 +0300 Message-Id: <1234641709058366@o5ump4w73ck7ppml.iva.yp-c.yandex.net> Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.10 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[yandex.ru,none]; R_DKIM_ALLOW(-0.20)[yandex.ru:s=mail]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:c00::/40]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[2a02:6b8:c02:900:1:45:d181:d500:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_FROM(0.00)[yandex.ru]; FREEMAIL_ENVFROM(0.00)[yandex.ru]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yandex.ru:dkim]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org,stable@freebsd.org]; DKIM_TRACE(0.00)[yandex.ru:+] X-Rspamd-Queue-Id: 4TkmC74f1Nz4v4D > I get the following from my own portable BD machine, from the lsusb command: > > Bus /dev/usb Device /dev/ugen0.10: ID 0e8d:1956 MediaTek Inc. Samsung SE-506 Portable BluRay Disc Writer > > There's also lots of "can't get device qualifier: Input/output error" with -v but it's not stopping the disk from working. I can run k3b and use it. > >> can't get debug descriptor: Input/output error. >> Does it mean that my USB hardware is broken? > > I'm not sure - it might be worthwhile subscribing & posting to freebsd-usb@ > > -- Thanks, void, I'll try to use freebsd-usb@. Regards, Serguey. From nobody Tue Feb 27 18:26:06 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkmCB03bxz5CBg4; Tue, 27 Feb 2024 18:26:14 +0000 (UTC) (envelope-from serguey-grigoriev@yandex.ru) Received: from forward500b.mail.yandex.net (forward500b.mail.yandex.net [IPv6:2a02:6b8:c02:900:1:45:d181:d500]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkmC74f1Nz4v4D; Tue, 27 Feb 2024 18:26:11 +0000 (UTC) (envelope-from serguey-grigoriev@yandex.ru) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yandex.ru header.s=mail header.b=nnP0miBU; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (mx1.freebsd.org: domain of serguey-grigoriev@yandex.ru designates 2a02:6b8:c02:900:1:45:d181:d500 as permitted sender) smtp.mailfrom=serguey-grigoriev@yandex.ru Received: from mail-nwsmtp-mxback-production-main-32.iva.yp-c.yandex.net (mail-nwsmtp-mxback-production-main-32.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:9497:0:640:2979:0]) by forward500b.mail.yandex.net (Yandex) with ESMTPS id 83337610FB; Tue, 27 Feb 2024 21:26:06 +0300 (MSK) Received: from mail.yandex.ru (2a02:6b8:c0c:bc83:0:640:2077:0 [2a02:6b8:c0c:bc83:0:640:2077:0]) by mail-nwsmtp-mxback-production-main-32.iva.yp-c.yandex.net (mxback/Yandex) with HTTP id vPdOF0AoLqM0-nHtBvhI3; Tue, 27 Feb 2024 21:26:06 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1709058366; bh=t96i187CztBDAY8R9tvjKRo9ssffl0cESGd0z57AcK0=; h=Message-Id:References:Date:Cc:Subject:In-Reply-To:To:From; b=nnP0miBUJ2ZnugZA+gI7Bmax8mdWmrANR1I3AW4+AmYqIFasKLnrAthsK9TpY0YHO tDr8Hi9d+MqCMJE7fYcQyhXEr7lw1efxukcVJLp12S/khLy7H3gRQvuHsUcY9EPLpD 9EwdlacCyOQv1wsappC9I+GWDezkS+EBQIXY9CXQ= Received: by o5ump4w73ck7ppml.iva.yp-c.yandex.net with HTTP; Tue, 27 Feb 2024 21:26:06 +0300 From: S.N.Grigoriev To: FreeBSD Stable Cc: stable@freebsd.org In-Reply-To: <6bf693a3-6fba-4bf3-a971-c71dbffd57b4@app.fastmail.com> References: <5813111708783942@wckjnuw7tehz7zgt.sas.yp-c.yandex.net> <3c8667a2-770c-4c35-9b33-572f43d9f1ea@app.fastmail.com> <4609701708855679@vpfns56j3qv6ec2k.vla.yp-c.yandex.net> <511954d9-74b4-45c3-b16e-342ad29751f7@app.fastmail.com> <478521708877670@z7oys7c4urhdnodv.sas.yp-c.yandex.net> <5a581a59-94fc-4476-97c4-4afac65c27bf@app.fastmail.com> <4874751708942864@omv3tj5zwvd3plqc.iva.yp-c.yandex.net> <6bf693a3-6fba-4bf3-a971-c71dbffd57b4@app.fastmail.com> Subject: Re: USB CD drive does not work with 14-Stable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Tue, 27 Feb 2024 21:26:06 +0300 Message-Id: <1234641709058366@o5ump4w73ck7ppml.iva.yp-c.yandex.net> Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.10 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[yandex.ru,none]; R_DKIM_ALLOW(-0.20)[yandex.ru:s=mail]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:c00::/40]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[2a02:6b8:c02:900:1:45:d181:d500:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_FROM(0.00)[yandex.ru]; FREEMAIL_ENVFROM(0.00)[yandex.ru]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yandex.ru:dkim]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org,stable@freebsd.org]; DKIM_TRACE(0.00)[yandex.ru:+] X-Rspamd-Queue-Id: 4TkmC74f1Nz4v4D > I get the following from my own portable BD machine, from the lsusb command: > > Bus /dev/usb Device /dev/ugen0.10: ID 0e8d:1956 MediaTek Inc. Samsung SE-506 Portable BluRay Disc Writer > > There's also lots of "can't get device qualifier: Input/output error" with -v but it's not stopping the disk from working. I can run k3b and use it. > >> can't get debug descriptor: Input/output error. >> Does it mean that my USB hardware is broken? > > I'm not sure - it might be worthwhile subscribing & posting to freebsd-usb@ > > -- Thanks, void, I'll try to use freebsd-usb@. Regards, Serguey. From eugen@grosbein.net Tue Feb 27 20:03:11 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkpMN2drzz5CL9D for ; Tue, 27 Feb 2024 20:03:28 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from mail.rdtc.ru (mail.rdtc.ru [62.231.190.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkpMM57Qmz3xJH for ; Tue, 27 Feb 2024 20:03:27 +0000 (UTC) (envelope-from eugen@grosbein.net) Authentication-Results: mx1.freebsd.org; none Received: by mail.rdtc.ru (RDTC Post Office Server, from userid 1000) id 0180C1CF49; Wed, 28 Feb 2024 03:03:17 +0700 (+07) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: egrosbein@rdtc.ru) by mail.rdtc.ru (RDTC Post Office Server) with ESMTPSA id 3D9311CF47 for ; Wed, 28 Feb 2024 03:03:17 +0700 (+07) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-stable@freebsd.org Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.17.1/8.17.1) with ESMTPS id 41RK3DNL072668 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 28 Feb 2024 03:03:13 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task To: "Edward Sanford Sutton, III" , freebsd-stable@freebsd.org References: From: Eugene Grosbein Message-ID: <8de04b21-f1cb-8dcc-9d2c-218cebea7c4e@grosbein.net> Date: Wed, 28 Feb 2024 03:03:11 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-0.2 required=5.0 tests=BAYES_00,HELO_MISC_IP, LOCAL_FROM,NICE_REPLY_A,SPF_PASS,T_DATE_IN_FUTURE_96_Q, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=4.0.0 X-Spam-Report: * -0.0 SPF_PASS SPF: sender matches SPF record * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 T_DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: * date * 2.6 LOCAL_FROM From my domains * -0.0 T_SCC_BODY_TEXT_LINE No description available. * -1.6 NICE_REPLY_A Looks like a legit reply (A) * 1.1 HELO_MISC_IP Looking for more Dynamic IP Relays X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on eg.sd.rdtc.ru X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29072, ipnet:62.231.184.0/21, country:RU] X-Rspamd-Queue-Id: 4TkpMM57Qmz3xJH 27.02.2024 7:24, Edward Sanford Sutton, III wrote: [skip] > More recently looked and see top showing threads+system processes shows > I have one core getting 100% cpu for kernel{arc_prune} which has 21.2 hours over a 2 hour 23 minute uptime. Against this bug, try these two: sysctl vfs.zfs.arc.meta_strategy=0 sysctl vfs.zfs.arc.meta_limit_percent=25 Note that in 14.0 ZFS changed a lot and no more has sysctl vfs.zfs.arc.meta_strategy, also it did not exist in FreeBSD 12.x, too. Eugene Grosbein From nobody Wed Feb 28 05:29:43 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tl2wy3p0Vz5CgZw for ; Wed, 28 Feb 2024 05:29:54 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tl2wx07Qdz4sws; Wed, 28 Feb 2024 05:29:52 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.17.1/8.17.1) with ESMTPS id 41S5TifO019388 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 28 Feb 2024 00:29:44 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.17.1/8.17.1/Submit) id 41S5ThbR019387; Wed, 28 Feb 2024 00:29:43 -0500 (EST) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26078.50375.679881.64018@hergotha.csail.mit.edu> Date: Wed, 28 Feb 2024 00:29:43 -0500 From: Garrett Wollman To: stable@freebsd.org Cc: rmacklem@freebsd.org Subject: 13-stable NFS server hang X-Mailer: VM 8.2.0b under 29.1 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Wed, 28 Feb 2024 00:29:44 -0500 (EST) X-Spam-Status: No, score=-0.8 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.86 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.959]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[wollman]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4Tl2wx07Qdz4sws Hi, all, We've had some complaints of NFS hanging at unpredictable intervals. Our NFS servers are running a 13-stable from last December, and tonight I sat in front of the monitor watching `nfsstat -dW`. I was able to clearly see that there were periods when NFS activity would drop *instantly* from 30,000 ops/s to flat zero, which would last for about 25 seconds before resuming exactly as it was before. I wrote a little awk script to watch for this happening and run `procstat -k` on the nfsd process, and I saw that all but two of the service threads were idle. The three nfsd threads that had non-idle kstacks were: PID TID COMM TDNAME KSTACK 997 108481 nfsd nfsd: master mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common 997 960918 nfsd nfsd: service mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 997 962232 nfsd nfsd: service mi_switch _cv_wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline I'm suspicious of two things: first, the copy_file_range RPC; second, the "master" nfsd thread is actually servicing an RPC which requires obtaining a lock. The "master" getting stuck while performing client RPCs is, I believe, the reason NFS service grinds to a halt when a client tries to write into a near-full filesystem, so this problem would be more evidence that the dispatching function should not be mixed with actual operations. I don't know what the clients are doing, but is it possible that nfsrvd_copy_file_range is holding a lock that is needed by one or both of the other two threads? Near-term I could change nfsrvd_copy_file_range to just unconditionally return NFSERR_NOTSUP and force the clients to fall back, but I figured I would ask if anyone else has seen this. -GAWollman From nobody Wed Feb 28 14:57:03 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlHWg5j5Nz5BkHf for ; Wed, 28 Feb 2024 14:57:19 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlHWg1wqnz4nD5; Wed, 28 Feb 2024 14:57:19 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1dc75972f25so47495585ad.1; Wed, 28 Feb 2024 06:57:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709132238; x=1709737038; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Th+b8W2snjUYShMVDKzdh+qGsUhoKWUyoBhvKJBMYzg=; b=EiLZUWC/6z3eUcfyOft0iNqhj3IiK1Gipoh6IS6HmeOJN+at5IoNYrpY1F02fmjTIH 2MochF8neYv2pA1MI8uH/qmyxoOANupGvkcbkjycs2qaEXdbhab4PGVyXdwoQjo75lyJ dpO0ki7G4tMm8B72QUpHmIk7V2Zbel5CLL5FTytoDUa+XcbA1hdh8iOQM7OEGxquebMO k6GB0spTCp9grueHnCjjURyO0a31iRYPisPgzqOaEK98IPyVBU3DE7iOV03fI4yqy1wY DuZC+M7QiB/NR5Zweo2dVqh39Hivi7dcXjeZUCBWn3YJMZWQbBatHlQT4MgvXfniaD2f 5kiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709132238; x=1709737038; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Th+b8W2snjUYShMVDKzdh+qGsUhoKWUyoBhvKJBMYzg=; b=oCWCZvh+e1IoDscT+uAOq2GVTzsH9CAR85qonDmfPNtez6wyaaG9895ap50l1rlZ64 ykTtSK1pLX/KV5uwhAytkwkNs3siTlfph3PevcSFTrtt5oSZGUlngNMgsVczLk36Bcsl qSYYyqU88Gj6weh7YUC1x0DGAgth2NJb2XYGmUlnIhl+6YnW9EK5mRdWARzijP9xeHIU 1KxGo226/n0qpwcDqHzjEdTNaVBKbSRaw6GMoxDH7f7MXAABbyaaL3pQYiLI6qYICeRj /BQVDO+2YnirEzbMtM6krbnCflQ+7r1hh5xe7R8svI/0PO1jalE7op4WbRAviMjh7NP0 HVig== X-Forwarded-Encrypted: i=1; AJvYcCVz0VpDea1bHKVFV6cLJ2kQBJXod3WOBN77mUwr4eTPBBa4p6geAC0WtT4VuX2b7TPIAcz7FMjXH64gLuWTz4z+5bTuUw== X-Gm-Message-State: AOJu0Yyk0eAm19pxfgTk1RB94le4nFDMQ57ktS77wOoZC/A6lBmzFfZV eWFtYkvuk9XOTruLEzXRC+n/OErV1xZI5V2bzvDgZKwVvPnSOMgtc0FFG/9K+ukl5G3FO0f2YP4 7Rb6FekywyYFTrH+0qyjUnD5ZMxwmLl0= X-Google-Smtp-Source: AGHT+IFin/Emfma56uB2TJLTFBc0Vl624TdBgmgHKgv1CZG8jMm0UvzIUl8IN67dVr0ne4ronyCvvWwZGaAPJ8L54WQ= X-Received: by 2002:a17:90b:3847:b0:29a:feba:abbd with SMTP id nl7-20020a17090b384700b0029afebaabbdmr787480pjb.0.1709132237690; Wed, 28 Feb 2024 06:57:17 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <26078.50375.679881.64018@hergotha.csail.mit.edu> In-Reply-To: <26078.50375.679881.64018@hergotha.csail.mit.edu> From: Rick Macklem Date: Wed, 28 Feb 2024 06:57:03 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Garrett Wollman Cc: stable@freebsd.org, rmacklem@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4TlHWg1wqnz4nD5 On Tue, Feb 27, 2024 at 9:30=E2=80=AFPM Garrett Wollman wrote: > > Hi, all, > > We've had some complaints of NFS hanging at unpredictable intervals. > Our NFS servers are running a 13-stable from last December, and > tonight I sat in front of the monitor watching `nfsstat -dW`. I was > able to clearly see that there were periods when NFS activity would > drop *instantly* from 30,000 ops/s to flat zero, which would last > for about 25 seconds before resuming exactly as it was before. > > I wrote a little awk script to watch for this happening and run > `procstat -k` on the nfsd process, and I saw that all but two of the > service threads were idle. The three nfsd threads that had non-idle > kstacks were: > > PID TID COMM TDNAME KSTACK > 997 108481 nfsd nfsd: master mi_switch sleepq_tim= edwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_r= un nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common My guess is that "master" is just a glitch here. The RPCs are all serviced by kernel threads. The "server" process is just the process these threads would normally be associated with. The "master" process only enters the kernel to add a new TCP socket for servicing. (ie. I don't know why procsta= t thought the kernel thread was associated with "master" instead of "server", but it shouldn't matter.) > 997 960918 nfsd nfsd: service mi_switch sleepq_tim= edwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfs= svc_program svc_run_internal svc_thread_start fork_exit fork_trampoline This one is a fresh mount (note the exchangeid) and this one needs an exclu= sive lock on the NFSv4 state. Put another way, it is trying to stop all other nfsd thread activity, so that it can add the new client. > 997 962232 nfsd nfsd: service mi_switch _cv_wait t= xg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_i= octl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE = vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_r= un_internal svc_thread_start fork_exit fork_trampoline Copy file range has some code in the generic function that limits the copy time to one second for a large copy. I had hoped this was sufficient to avoid Copy from "hogging" the server. My guess is that the 1 second limit is not work= ing. This may be because the ZFS function does not return for a long time. When I wrote the code, ZFS did not have its own VOP_COPY_FILE_RANGE(). (There is a special "kernel only" flag that needs to be set in the argument for vn_generic_copy_file_range() that ZFS might not be doing. I'll look la= ter to-day. However, it will need to be fixed for the case where it does not call vn_generic_copy_file_range().) When I first did the code, I limited the time it spent in VOP_COPY_FILE_RAN= GE() by clipping the size of the Copy, but the problem is that it is hard to estimate how large a transfer can take without knowing any details about the storage. I might have to go back to this approach. > > I'm suspicious of two things: first, the copy_file_range RPC; second, > the "master" nfsd thread is actually servicing an RPC which requires > obtaining a lock. The "master" getting stuck while performing client > RPCs is, I believe, the reason NFS service grinds to a halt when a > client tries to write into a near-full filesystem, so this problem > would be more evidence that the dispatching function should not be > mixed with actual operations. I don't know what the clients are > doing, but is it possible that nfsrvd_copy_file_range is holding a > lock that is needed by one or both of the other two threads? Sort of. My guess is that ZFS's VOP_COPY_FILE_RANGE() is taking a long time doing a large copy while holding the shared lock on the nfsd. Meanwhile the second thread is trying to acquire an exclusive lock on the nfsd so that it can add the new client. This means that the other nfsd threads will slowly get blocked iuntil they all release the shared lock. > > Near-term I could change nfsrvd_copy_file_range to just > unconditionally return NFSERR_NOTSUP and force the clients to fall > back, but I figured I would ask if anyone else has seen this. Yes, I think that is what you will need to do to avoid this. Thanks for reporting it. I have some work to do, I need to think of how ZFS's VOP_COPY_FILE_RANGE() can be limited so that it does not "hog" the server. rick > > -GAWollman > > From nobody Wed Feb 28 18:38:24 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlNQp5Plfz5C5RW for ; Wed, 28 Feb 2024 18:38:26 +0000 (UTC) (envelope-from leres@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlNQp4dPNz4FLr for ; Wed, 28 Feb 2024 18:38:26 +0000 (UTC) (envelope-from leres@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709145506; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZzjAr875dz/LRTXIhijJ/TfJR5I7nfNAdxaHlbfLv4w=; b=yBWsMfLYw+/dM8NOKatb9/qj4xKj7fv1LQHEfw6KNEEs1/68WWJ+1Df/oAGmWQHayYN7km rzLLcQiPu6y97K2T5tc/r4f+uAryDp6DXMeqfIRrgUDF+P5pqaBCKm5w1H5lYhGmpFSUxX LW/TIfGCRNSTXwEfWJEIOP7cDt8kY4OzoReoH/QOQYqIvtnhHcluvVhS13NqfUm3Fje7MS ggtsBuiRFP6kbS4c0nWryKr53c1ioJb/olrSj2CyDX8hgo6Pp6v/9ZDby8Q38zMYW7FnZm W1mBRzxUyLHotq7sFtHrVirSM1ZdAJnR5zkbK6Uv1C86Ufw7JYI2EtBKdHfPXw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1709145506; a=rsa-sha256; cv=none; b=O7BQQFMhjtrjn5FM7fgopNdJPGRhU/s2LEz87R8zb2rgTRTZ+gQAosIZOMT3KDmNrG53MD mO3xUP/g2OfwnwtCpexDmHpZ7LGRoccpoQSWajyp25SRyibTMJ65PCPDT/SyfECq2l1YyY +VHInijWIbkUDrazy77hiVmseGIofk/1Vzcvet7fFl/2Y005BuxZgdGGPQfEJ69qTwLLgw VcGDPYwRFjJfZh7qVxJjttA4/f2QA2B/0Ncy+w7Nfw/IoW3A79OfAh6lfdsQDVk4z1EhXW 7393oBC6BJYtcOZYyud3PpmAsd1aZ+3O6VKUP9nNEHR4us/q9F5Dvr1dvx2GNw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1709145506; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZzjAr875dz/LRTXIhijJ/TfJR5I7nfNAdxaHlbfLv4w=; b=RbFf8h7vJAitwYQIH6q8A47KpZSsQrrmBajUe5AIkawWz03bjjaJ7tV9XHYw5n1UngsRNJ uIfCeBrzNsTWc9OWTltluRxYxr77EEoDyK+9GU+gGBruIOJqYm+N9xyf48y3f/7wa1AY4x ElpgbvjWuDHjGlJcwXCZUpU9DDD9+uNygbrhS4y6S7JTG5DPrjx8+VQn6gPMY0txYijiqw vdDm7/NJlvWNm7UwUy12AlWlKsFwo3mtd++NC3r8uAxlZfZ84EttM/durf5BJI7HUrBQ2U LJfo8EE8XK70g0dDOwFKl786MyMXm7WiXA9L/rNr0ZwLhxoLKOQJe+vOLWDexg== Received: from [IPV6:fd:1965::2] (unknown [IPv6:2600:1700:ab1b:6800:2e0:edff:fece:8f27]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: leres) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TlNQp2LYbz1FTp for ; Wed, 28 Feb 2024 18:38:26 +0000 (UTC) (envelope-from leres@freebsd.org) Message-ID: Date: Wed, 28 Feb 2024 10:38:24 -0800 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: net/libpfctl: needs to be updated for 13.3 Content-Language: en-US From: Craig Leres To: freebsd-stable@freebsd.org References: <80a7ec56-0e9d-4f26-bbe1-427c8672045c@freebsd.org> In-Reply-To: <80a7ec56-0e9d-4f26-bbe1-427c8672045c@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/26/24 17:04, Craig Leres wrote: > While building a bunch of ports under 13.3-RC1 I find that libpfctl does > not (build). I freely admit I do not understand how this port works; I > started to create a PR but I don't have the first idea on how the > distfile is generated. > > Meanwhile 1136 other ports built for me without issue. > >         Craig > > => libpfctl-13.3_2.tar.gz is not in /usr/ports/net/libpfctl/distinfo. > => Either /usr/ports/net/libpfctl/distinfo is out of date, or > => libpfctl-13.3_2.tar.gz is spelled incorrectly. > *** Error code 1 Looks like this was addressed yesterday via 0a5b676fc982. Craig From nobody Thu Feb 29 00:04:41 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlWgY1dzVz5CgP3 for ; Thu, 29 Feb 2024 00:04:57 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlWgX6qZmz51cC; Thu, 29 Feb 2024 00:04:56 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-29a61872f4eso189454a91.2; Wed, 28 Feb 2024 16:04:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709165095; x=1709769895; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=XBSSjGcbvpox77B9sWvMW2EwHCgeaukw/lJku+7rabI=; b=G/9Njue1gpB5kf82TFnC6a+VIQy1ZVPo+xncvpU9Wdp5xV5syca2CI1N/HrAr62rD9 106xT/3GAMcS7JGz1AXzqysZoTaN6RLGASG2+us1TgQyCxkDejnPf4+vr8ebZRHRl3z8 LIHmZPM9uWsXwFjSF6dMHno15jeS8wxBHHfMA9pRV90jDH4WMirg+O5zPtjh4pLsIcsC fu78YoeftZu+Y523mpY3PWBfusmGF7E2me0ni2jrs8olM0wpQ0f9YITEadXMRRZOqcM7 iqXbCzI3YVLOcSASyB5ChU3eM9II+0W1YX37VVa1l/pHtEk4eUiLefgbktUAZyqmAugt sYfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709165095; x=1709769895; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=XBSSjGcbvpox77B9sWvMW2EwHCgeaukw/lJku+7rabI=; b=sjqQZ57BXwjlxi3mhtbvmV0p1zbfZRDevs+vlZwNd/9HOOTQyRjrM9gcmmBh52+p4h w+3O9mXKflL5subyqMiNYlbC2vj72ca77BIZZWBz1xSJe5tcLQ5gHbw6G9zOTZtCdFVu ARIOTyHAQddxisGZj/9Co8Rhaa3cFp4zmuRvGv3u98jJ2rwKOcqBHrgRlIu2cgLue9rm XfY81rrg70Ldu1vhLIeIW+QK+kcwO7cQ/RaCx5G84Wx5fdMjcs2nZRj+s21qHdw3YV2r EHefpeI24PLdkcD2U/yQ6UYH/l5SN0MqEMjenWc21gykSLgYAlj+J53YXCgDbeCS6k5D BAyQ== X-Forwarded-Encrypted: i=1; AJvYcCX6piSOSL8mIf3iWdWcbqIn74ReWHWfjjXT7rZhalpULMqvnxxOD0XQbXsutz7DIkbwswbHM20Ty90gCQEIHZyM5BJopA== X-Gm-Message-State: AOJu0YxbmnaLUcjYmAhFkjpQ0dmqAcCfRbwgptr/9FR8cHOhR+q8692l bi1NKatZKDIRkH/EBbO/9Yr0mH/iGa00GnwbZTOG3jJKNx9AkbCZeH96OanalNzT4ENwJe59C4y UyDIB4BCHt2MQBNF7/v3Uz6gT0D3dTqQ= X-Google-Smtp-Source: AGHT+IHHR5cz0PxPtK6kddqp8KkSmxuc+U2DWRTQAJW1qVjhncvr259tGvatqvLD/D5Od6gduZbhU3/QKmDpzo4+700= X-Received: by 2002:a17:90a:bc84:b0:29a:c4a3:ca0a with SMTP id x4-20020a17090abc8400b0029ac4a3ca0amr851859pjr.18.1709165095491; Wed, 28 Feb 2024 16:04:55 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <26078.50375.679881.64018@hergotha.csail.mit.edu> In-Reply-To: <26078.50375.679881.64018@hergotha.csail.mit.edu> From: Rick Macklem Date: Wed, 28 Feb 2024 16:04:41 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Garrett Wollman Cc: stable@freebsd.org, rmacklem@freebsd.org Content-Type: multipart/mixed; boundary="000000000000613a6806127a02f4" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Queue-Id: 4TlWgX6qZmz51cC --000000000000613a6806127a02f4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Feb 27, 2024 at 9:30=E2=80=AFPM Garrett Wollman wrote: > > Hi, all, > > We've had some complaints of NFS hanging at unpredictable intervals. > Our NFS servers are running a 13-stable from last December, and > tonight I sat in front of the monitor watching `nfsstat -dW`. I was > able to clearly see that there were periods when NFS activity would > drop *instantly* from 30,000 ops/s to flat zero, which would last > for about 25 seconds before resuming exactly as it was before. > > I wrote a little awk script to watch for this happening and run > `procstat -k` on the nfsd process, and I saw that all but two of the > service threads were idle. The three nfsd threads that had non-idle > kstacks were: > > PID TID COMM TDNAME KSTACK > 997 108481 nfsd nfsd: master mi_switch sleepq_tim= edwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_r= un nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > 997 960918 nfsd nfsd: service mi_switch sleepq_tim= edwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfs= svc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > 997 962232 nfsd nfsd: service mi_switch _cv_wait t= xg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_i= octl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE = vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_r= un_internal svc_thread_start fork_exit fork_trampoline > > I'm suspicious of two things: first, the copy_file_range RPC; second, > the "master" nfsd thread is actually servicing an RPC which requires > obtaining a lock. The "master" getting stuck while performing client > RPCs is, I believe, the reason NFS service grinds to a halt when a > client tries to write into a near-full filesystem, so this problem > would be more evidence that the dispatching function should not be > mixed with actual operations. I don't know what the clients are > doing, but is it possible that nfsrvd_copy_file_range is holding a > lock that is needed by one or both of the other two threads? > > Near-term I could change nfsrvd_copy_file_range to just > unconditionally return NFSERR_NOTSUP and force the clients to fall > back, but I figured I would ask if anyone else has seen this. I have attached a little patch that should limit the server's Copy size to vfs.nfsd.maxcopyrange (default of 10Mbytes). Hopefully this makes sure that the Copy does not take too long. You could try this instead of disabling Copy. It would be nice to know if this is suffciient? (If not, I'll probably add a sysctl to disable Copy.) rick > > -GAWollman > > --000000000000613a6806127a02f4 Content-Type: application/octet-stream; name="copylen.patch" Content-Disposition: attachment; filename="copylen.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_lt6gr9sk0 LS0tIHN5cy9mcy9uZnNzZXJ2ZXIvbmZzX25mc2RzZXJ2LmMuY29weWxlbgkyMDI0LTAyLTI4IDE1 OjM1OjQ3LjcwMDUzMTAwMCAtMDgwMAorKysgc3lzL2ZzL25mc3NlcnZlci9uZnNfbmZzZHNlcnYu YwkyMDI0LTAyLTI4IDE1OjQxOjMzLjcyMzAyMjAwMCAtMDgwMApAQCAtOTksNiArOTksOSBAQCBT WVNDVExfQk9PTChfdmZzX25mc2QsIE9JRF9BVVRPLCBlbmFibGVfdjQyYWxsb2NhdGUsIEMKIFNZ U0NUTF9CT09MKF92ZnNfbmZzZCwgT0lEX0FVVE8sIGVuYWJsZV92NDJhbGxvY2F0ZSwgQ1RMRkxB R19SVywKICAgICAmbmZzcnZfZG9hbGxvY2F0ZSwgMCwKICAgICAiRW5hYmxlIE5GU3Y0LjIgQWxs b2NhdGUgb3BlcmF0aW9uIik7CitzdGF0aWMgdWludDY0X3QgbmZzcnZfbWF4Y29weXJhbmdlID0g MTAgKiAxMDI0ICogMTAyNDsKK1NZU0NUTF9VNjQoX3Zmc19uZnNkLCBPSURfQVVUTywgbWF4Y29w eXJhbmdlLCBDVExGTEFHX1JXLAorICAgICZuZnNydl9tYXhjb3B5cmFuZ2UsIDAsICJNYXggc2l6 ZSBvZiBhIENvcHkgc28gUlBDIHRpbWVzIHJlYXNvbmFibGUiKTsKIAogLyoKICAqIFRoaXMgbGlz dCBkZWZpbmVzIHRoZSBHU1MgbWVjaGFuaXNtcyBzdXBwb3J0ZWQuCkBAIC01Nzc4LDcgKzU3ODEs MTUgQEAgbmZzcnZkX2NvcHlfZmlsZV9yYW5nZShzdHJ1Y3QgbmZzcnZfZGVzY3JpcHQgKm5kLCBf X3VuCiAJCQluZC0+bmRfcmVwc3RhdCA9IGVycm9yOwogCX0KIAotCXhmZXIgPSBsZW47CisJLyoK KwkgKiBEbyB0aGUgYWN0dWFsIGNvcHkgdG8gYW4gdXBwZXIgbGltaXQgb2YgdmZzLm5mc2QubWF4 Y29weXJhbmdlLgorCSAqIFRoaXMgbGltaXQgaXMgYXBwbGllZCB0byBlbnN1cmUgdGhhdCB0aGUg UlBDIHJlcGxpZXMgaW4gYQorCSAqIHJlYXNvbmFibGUgdGltZS4KKwkgKi8KKwlpZiAobGVuID4g bmZzcnZfbWF4Y29weXJhbmdlKQorCQl4ZmVyID0gbmZzcnZfbWF4Y29weXJhbmdlOworCWVsc2UK KwkJeGZlciA9IGxlbjsKIAlpZiAobmQtPm5kX3JlcHN0YXQgPT0gMCkgewogCQluZC0+bmRfcmVw c3RhdCA9IHZuX2NvcHlfZmlsZV9yYW5nZSh2cCwgJmlub2ZmLCB0b3ZwLCAmb3V0b2ZmLAogCQkg ICAgJnhmZXIsIENPUFlfRklMRV9SQU5HRV9USU1FTzFTRUMsIG5kLT5uZF9jcmVkLCBuZC0+bmRf Y3JlZCwK --000000000000613a6806127a02f4-- From nobody Thu Feb 29 13:40:05 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tlstf5KHMz5CD5R for ; Thu, 29 Feb 2024 13:45:46 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tlstc6dWSz4cbK for ; Thu, 29 Feb 2024 13:45:44 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of li-fbsd@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=li-fbsd@citylink.dinoex.sub.org; arc=pass ("uucp.dinoex.org:s=M20221114:i=1") Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.18.1/8.18.1) with ESMTPS id 41TDj6vH046799 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 29 Feb 2024 14:45:07 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) ARC-Seal: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1709214309; cv=none; b=mmlZ40tEd+c3llHral9P3a6yYAJZUG1fI5cNkoBXtxP2+RDnHfNqBkR0WVmqsOWGLVPHRHHXopvTNkha6Tv8zZkJQ2YrstCLgK1cEbKJKqGKUv3TfwvBBVnkfXta5BNTvydH6dB+vBMcgJhAPxWD9OiNDmrjH9XOqdPfPy5lInI= ARC-Message-Signature: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1709214309; c=relaxed/simple; bh=rI1yjz/H6DS+qXi5EqAF0XQIDrHmOj/tvD0248xAtVM=; h=Received:Received:Received:X-Authentication-Warning:From: X-Newsgroups:Subject:Date:Message-ID:References:Injection-Date: Injection-Info:User-Agent:To:X-Milter:X-Greylist; b=VwFtMBO8ag75ucEZgipLbG2d+T4BGMxf0sq18HwTE8TR6tSm1bAZk+I3HL5aVAh9xUMcVrFq6z89xAa/Glz/A52xRGz43A/eVdgx7GUWkgV3gqYrkWSTLpyd2qAY9E7F2VwIxC/zyjV8HOqXPLTDqAx7Ndybj8MLHAoCmnDjyNU= ARC-Authentication-Results: i=1; uucp.dinoex.org X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.18.1/8.18.1/Submit) with UUCP id 41TDj67f046798 for freebsd-stable@freebsd.org; Thu, 29 Feb 2024 14:45:06 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from admn.intra.daemon.contact (localhost [127.0.0.1]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 41TDeHif011743 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 29 Feb 2024 14:40:17 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from intra.daemon.contact (news@localhost) by admn.intra.daemon.contact (8.17.1/8.17.1/Submit) with NNTP id 41TDe5pN011567 for freebsd-stable@freebsd.org; Thu, 29 Feb 2024 14:40:05 +0100 (CET) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-Authentication-Warning: admn.intra.daemon.contact: news set sender to li-fbsd@citylink.dinoex.sub.org using -f From: "Peter 'PMc' Much" X-Newsgroups: m2n.fbsd.stable Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task Date: Thu, 29 Feb 2024 13:40:05 -0000 (UTC) Message-ID: References: <5375.1708993627.m2n@admn.intra.daemon.contact> Injection-Date: Thu, 29 Feb 2024 13:40:05 -0000 (UTC) Injection-Info: admn.intra.daemon.contact; logging-data="90586"; mail-complaints-to="usenet@citylink.dinoex.sub.org" User-Agent: slrn/1.0.3 (FreeBSD) To: freebsd-stable@freebsd.org X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Thu, 29 Feb 2024 14:45:09 +0100 (CET) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; ARC_ALLOW(-1.00)[uucp.dinoex.org:s=M20221114:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; FORGED_SENDER(0.30)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; HAS_XAW(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; FROM_NEQ_ENVFROM(0.00)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[sub.org]; RCVD_TLS_LAST(0.00)[]; TO_DN_NONE(0.00)[] X-Rspamd-Queue-Id: 4Tlstc6dWSz4cbK List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org On 2024-02-27, Edward Sanford Sutton, III wrote: > More recently looked and see top showing threads+system processes > shows I have one core getting 100% cpu for kernel{arc_prune} which has > 21.2 hours over a 2 hour 23 minute uptime. Ack. > I started looking to see if > https://www.freebsd.org/security/advisories/FreeBSD-EN-23:18.openzfs.asc > was available as a fix for 13 but it is not (and doesn't quite sound > like it was supposed to apply to this issue). Would a kernel thread time > at 100% cpu for only 1 core explain the system becoming unusually > unresponsive? That depends. This arc_prune issue does usually go alongside with some other kernel thread (vm-whatever) also blocking, so you have two cores busy. How many remain? There is an updated patch in the PR 275594 (5 pieces), that works for 13.3; I have it installed, and only with that I am able to build gcc12 - otherwise the system would just OOM-crash (vm.pageout_oom_seq=5120 does not help with this). I didn't see any lagging behaviour, but then, I have 20 vCore. cheerio, PMc From nobody Thu Feb 29 16:02:42 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tlwx02Hs0z5CR7M for ; Thu, 29 Feb 2024 16:03:00 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-19.consmr.mail.gq1.yahoo.com (sonic313-19.consmr.mail.gq1.yahoo.com [98.137.65.82]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tlwwz2yxZz4sVh for ; Thu, 29 Feb 2024 16:02:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=XzivtZN6; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.82 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709222575; bh=pcitlcnWOabPzg+WzS1XzMWnfLbV4lDj1xvwUifkkpc=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=XzivtZN6sE+0uUxu6v+NECdVHbwqfqGzmXgQu/J9B6hWlnW0XvRy8XPZ5gInbwlIT7DYjUcdkhQ8QMr3m5PuNOE9okkrqRO4HPdmUiR09ci7hqrVNGRYOCsXErbPNNikYnGYrRS7TnCWuvaokK5mxhswBsSlMx5cnXQZmmPtV2H6KD8hGdbwdrAc8TtFr/V1znMlr0eayY4PE7aZXGx07KOS0CP5W5BAji7UF8sVHX+oLYCza3mGxHwztkPc/wGdY9ReqboURQaBA/EvYjW4V+CaVAnSaV7ZFc1lNVwWwn7jfU+p7vSauIXjrdMPZsMUX7qo/SZm79PcKuggBYKFfw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709222575; bh=rFnEULDbIP1+MreXgodn5ai/GwN7hpHgy2CLt93pLjz=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Zsqb3MgNab8Bpf2oXgn0tj7tZyDpQvdJvBI2uwuE5EaVp/pkZ8viNWUV+QrwhXBw9pBM/wTodJ0dMyVsPrrcz3Pyq4NpRRQZT6VgNQ+4vGNDO8Ayux7/cVxwJGcDIeQTjIV0rsA7/uwqi9PjmHc4bXT9+k3oOK792oGfupdjI8PfaCQLJyBD6oVPRJ7yvTEAoawIkeOrbVsIaSTrVksU7lUfSYo3oUoEk3eSfZNo1PSf3VC/RkvKQ1MjtDIdWnbyn8pj0oEp1xDdqCB1WRAlMUlwl4qshM8+crEsReeCgRPSNVRNyfKpc4ifBJuFqv+kd0RomBF8P9TuE5rWPl/avA== X-YMail-OSG: iKbbwCEVM1mRHJFDoC0Ps51Y.V1zX72_wT_VmDOGB98niSXl2xz3VFyT7oHZYQA 0wpxlMxiy0OdfJRbL25cqyIxssZzU6kMKmEUys8mGbIEWbdTL4VNjGqhYwLS_hc6jNJBRQrMq_qi vooY.b_8R2XkqWr4T6PF6KySmQcbWvzJeQVJ9Ho5iXGAmQ_d7vhB.mvf5qgGoFT2aXsD5ZFZySG4 ybWksdd5bsJ4OiyYhBQiHuiFezhIRCHHUg.UJ43.w_HChqtjmy2GIJk6DkrYtYICU2MyyZNpWNsH l0LbyQWxdW0FYgZFBjbIlAcDGDj7BJqL0D6Dv35ALoR.ilPtySjmoAD4fyMY1.shGKQUoAzneIAO NkVA7xv_58jj99LpzMXd9xzHO9ex6dm5McyVnFfCqZMTS.VzY.HA5LUTB881GaETpKBjVLflF3Gh 1kcsWcwrPtDNqH9AdgdFtNii3IPYJGzZbW16DLtiSQHHOgc2wv31_56fxBolO_RHmZDS7duO_.qf 6sfaUpJ3t39lwlXAS3vIuc4kcde8xketiJG8Z.f4HVFtz7mdZIEAUQXLqr3KI2znWCUNOB46JEN7 sy4YW.oHihRG2FNNluWXrGHzGoIwu_O1CkY1SXDuV7L_Jk63ol0qWNtABcmWi5WpGpof_8xlIyk8 j2.Qvlunv9lDkxnzrhL0Uhfb4ePb2r93zFhFUZbg7atUFX3gISfff_fuijlNUTN4MrfKlL6Mk3.I eHHXLjxvyMtkeGPJvs4srPbnDxX32d7j5pZjRcthFPaZZcoWJu6Bao4kPH8GrhvD8sGCHoYQWuIA 9WId1.WtBWpkvb2sCogXfT1QO2d5jbI0LTQvBFHUggouUut3nBsM1hmBvftWQPhSFX_CNUljLUWc KzSOqSNsTv_4XdjzpVpHGmMEkdYGYTSE_VlLn02xI5aB4fhOFGA.08q.mNeHoZ_qZCK_WUWujuFW zUYZx5z0r5j0d4pTxlcyhwGXQZDCWsj_cr.Gs1I7HfWSAmDbEUNSsI4ofC6K4fftVUwIHvmRyHom _35J0kX2SYEDGTGBEOYTUNSXlFu2cJhU0GM9JzMqZSHOA3U3mc3oiZZDQfhTGA0.N1iIcTRxd8PF 0MNtrAG_sX9kWCXAGHXu80cP5hDr4QWK7rUo0GBr3zmSLPL4MB.ovn9F4t0zJ_Zlv1.3.D_kyksV eaZLYNydCd_d51CtNm.vey4VN8tgGAeIc5dhzMLPE5W2HfnPnlUcO0IvuLnCWmqmpf9riD8dtpg1 QupDHxy1g1OoRbyKqCEvPf6j0LX.Azxe.tljCm8FouV_YJ8aS7KsoaxcXM5plwyEknZ48I2OwMtV mddZ1sA03Rs6Bf1oNkB.kkewxd1fO91zbCAmrtRS0KuWw_nt3l.aE0kYXrY3Z3Y.LZfkxLMKsQOi cuXDKQ4IG7uEUS9xmWMpeVE7MV5Lwdo2eduSeCOny72pNrMSmvTBJmoVJiPEdGNZVbN6dax.HoQw vHWI.zNyG5YSarRelxWE9DxEWjTg7kKd1nCAFjrRrcfeLUZII3pzNcsGi5KCL7zZ.MVFr.Jce3jS Ul3zj1j9yxub4pDFYxHWj7ihliVGpk1RvXgT0ntY3aNdbS1ugY8Uc83j61XgY5gDpf1WSkNhu8Uh IsJ77wPCmmS_eRo0VuqmE7CePzNn4a4IEcGooFOx1JaIxkWArwn0zNznzCcOTC_30y8cMLXxu0jY q9HWWKe7s0a2sHpeY7aYATlY3q5FTCXWroMitHS5e5oN4eHnHmbWmA4eaSNX3VGIvvWBTxKcW5zs XHf1QICLnJxaYu44bZG99eQYZoStSboZ_BcKdCmsKmtnoNLpNbNLo4YzMshQCRsJUkWny.PIEhF0 .84f2iYaaYo.k_TwjCSlgdFzNfPhpOV9WnD4RNtBXqptOMvf2U.wg1d.reDXi.dh7LmWiM_BiCWI I6wOjH1gq4V35vCroeUrSUyupz63STJiMLfvWQZ7R6lANFEzTyIHfgyhWjzbQgCHXFkRcU0riAwX ROoLniK6dCzkT0OVx_whaHCEJSkLwaZXSgkfgf6uCzYFwq_8ExJ5eeB2rpJ83SJUNARERpUa18TA wY4WmdtMT4BAzRRu.wF5y_HdNdPVZRjkvFUDzxAcxDkfiGcqcYF0_judxNiTO5tzt_M71ACg51v6 zyHJZUibNZGiQcJ1vjw1vh1mR8ujkzbrVlKUpIOwBO7Q2QwYn5OOuN3QL2sz7FqCfyifESEjR09N 92MYhn.NFL0yB8uTX4CkK1MQGcaTtEi70Atn9UIOdLhl2izZlYJzDzsrSsagB.ooMMKAmFpWMApf rWOJLYQ-- X-Sonic-MF: X-Sonic-ID: 63b3f007-7d3c-459d-858c-c2605ba7a7e1 Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Thu, 29 Feb 2024 16:02:55 +0000 Received: by hermes--production-gq1-5c57879fdf-vxz7c (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 18d7993375fc4c9e88db11713d51fd9b; Thu, 29 Feb 2024 16:02:54 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task Message-Id: <426089C7-A15C-4B04-BC47-D1F77089C492@yahoo.com> Date: Thu, 29 Feb 2024 08:02:42 -0800 Cc: "Edward Sanford Sutton, III" To: Peter , FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3774.400.31) References: <426089C7-A15C-4B04-BC47-D1F77089C492.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_CC(0.00)[hotmail.com]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[yahoo.com:+]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.82:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.82:from] X-Rspamd-Queue-Id: 4Tlwwz2yxZz4sVh Peter 'PMc' Much wrote on Date: Thu, 29 Feb 2024 13:40:05 UTC : > On 2024-02-27, Edward Sanford Sutton, III wrote: > > More recently looked and see top showing threads+system processes > > shows I have one core getting 100% cpu for kernel{arc_prune} which has > > 21.2 hours over a 2 hour 23 minute uptime. > > Ack. > > > I started looking to see if > > https://www.freebsd.org/security/advisories/FreeBSD-EN-23:18.openzfs.asc > > was available as a fix for 13 but it is not (and doesn't quite sound > > like it was supposed to apply to this issue). Would a kernel thread time > > at 100% cpu for only 1 core explain the system becoming unusually > > unresponsive? > > That depends. This arc_prune issue does usually go alongside with some > other kernel thread (vm-whatever) also blocking, so you have two cores > busy. How many remain? > > There is an updated patch in the PR 275594 (5 pieces), that works for > 13.3; I have it installed, and only with that I am able to build gcc12 > - otherwise the system would just OOM-crash (vm.pageout_oom_seq=5120 > does not help with this). The kernel has multiple, distinct OOM messages. Which type are you seeing? : "failed to reclaim memory" "a thread waited too long to allocate a page" "swblk or swpctrie zone exhausted" "unknown OOM reason %d" Also, but only for boot verbose: "proc %d (%s) failed to alloc page on fault, starting OOM\n" vm.pageout_oom_seq is specific to delaying just: "failed to reclaim memory" === Mark Millard marklmi at yahoo.com From nobody Thu Feb 29 16:06:40 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tlx1W66xzz5CRn5 for ; Thu, 29 Feb 2024 16:06:55 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-21.consmr.mail.gq1.yahoo.com (sonic310-21.consmr.mail.gq1.yahoo.com [98.137.69.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tlx1V5Q5Pz4tgM for ; Thu, 29 Feb 2024 16:06:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=eAQiknNE; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709222812; bh=VF9uaQb6lg3t2DMmre1IjQovhgfyZTb/Hj+spZ/+tDs=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=eAQiknNEXPgifMmQD0Q+9iN1ijpSVjuwsIinqOxmtfu8C0Wtj2Jb1+HqgvHJMiom/plbfQerwEgUzD5DU1BocNH8QdEnfl4ZT5OxVD1ZyYgC9/wzOQMzwgFT/lcK96VZsyznTvyLKQMw34wCY9edvitdsInkKsaUJQhg5X4lh58DDWu6UCOonjTOpXAUgwY4oGYtuHWlUij6J8h7P6NIkLMAYI+e0M4DKIW3NqlT1q+IxitDe+TDv5T7JBw3UDiQp6A45WP0swIJLqrgrkknucXov98PjKUMscf0t4dDYwU/XR4JcEGkKiW7rMpNEtk7whC6M575VOLPgnyMbNbyvQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709222812; bh=bRNRJzznRzbiiG4Sd+2HaLQLknPXWfY6frM7zt2nly7=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=T09TTUhnFEmYaBGlqPKFwcOpI9xlQi5YA6wziDZvgOU+DX5KYiIlZS/C7nT4IicPxRbtid2axnkJ5de229gsnf4vJ6+ns5oWpJtaiNsfqN95gz/tmW4NPH84ppLtcHodKhxxfOCpshhS+JLA/JSoZdQFUkAK1Jn4Fuc/f5wC5+Ny056Vkc5vb8RPoYiEbpgc56JAaozodwtPhPrX8fwd7u8tg6uUlje4ylpQ4TRWCkEa714lKX1UKqOtDwc54FGdd0L3kc6udP5qntmWwv1eWg3jxV5Bt9r6XCy8wOxUV3BCE5ScjymuU4M9/0gFm4aD+yukYBIO7+XQJR2RVUuqeg== X-YMail-OSG: SMG10IIVM1k9uvYUN9Ueu8eDqu0zJs7u4F6DZXvaqIQTcfaiLTL0RK2u51qO5UG X9HDwEve5Q1SFWhr_1_7C6uEbnRBFfZF1AICbqvUrdPMaoAeuhc3pN8Mh7h.uCZSrd0xJu8Gwqj6 i5qt_.85P5fuNKF_VrwYS2u6wnqRLvJ0CI9Qqd9wz3.heUg3yijOYe3iepdL3S1nVAGnFEkiFZK7 HkETdWol4jRMY0zlADq.99huWycVADzSzidFjeFx9XdCaBjhls4Hy21yBPSWJB5c63.fvfmvU7cd z2dzFJUyzL8Wdvd.O98r.UndnGPrjS70Yb86ywPlUnRPZm277C0IUR.mQs8wuALJLLQPgB24IoeR 7C7b6hsfuZIf0PCiDLGJip4UFTVCjcgUhc6As2AR7Y4pH5n0d0QYLToztmNiK_fcxfBiVSf9AUds NULF8X9Q2BN39cO_VPpxL.Xg1GJg2kvHnTWklAxUFLta1gLDBZE_f2Wofe8_MFcU1E2uy6kdgKo0 WdtX.VwPlTQblgOYiGf3T2IejrWU2nqzAosWPXYKytIRLHnu7ZZgGs07fsARelffsLqOKIyD2AuX 4ltA8aVhb5P3149BiRVJJocVVg702y0WrprQl2AyfpY_dFAY6t3YBYi7cmyysDdCJsI3glzk.Rgw bTREZ_LL1.k3bP14tNMG5c0v0dVF4sLfPUDpc0YnLOJBO6IhNK_8dSRWyTPIWRQUhbH1oxuxD2EH 0bzQ73JUTJBlgTCoBmgLw_kX7x.LpXI_5DhMyty4YkNRifstw49i5CxbWGUfbmYhrPat8JWOmK7f r6h4cvgs5XV65FjGJBd6isvhTJ4dbr5ajNL5yuXCDnYm0VD4RI3yI5CJjuvdnDzju9kKPNcXbheL dQO5CSK0_dQhNHyULTUCasPJ4xACt4vx4qoBqYw9SK7U20tHsZMSzvaNQlX5MsvNaFyo1xI8moUg XrxUXGBp6zmP7MQASMvUUzoYSSjPQt2QMvLhFz5pxfVCVzcKJ.3sy3QBwDh5CwWJpMas25ITUsJ. PIZkH2TuRt0z4FarDzI8W8jcXtcdgrW9VGEo3Lbo.bQ.4tOhiMV10WiDr6cCEobOO__jhjQAJO9Q VkbhGrZAoFWE.W1zScAtXAyygqmzxAe2ZyP0M6gMzHvbJ4M0D3BbtnoSd9MXQMkXb85gDsbJg4LD H.VrYnr9q8NvBAzq8E1SkP1OQn4QYL48OPpYSRdztlAHm21JA0fP2Y2X_Ab9Xg0jh32CA1kbqmWd ZnVp4OfnswrGyUULrd3k.cQnr_xGmojLHmf1CY.hTWCVT4ABKcMjHISndgPAR1SrvH8X6aspesiQ _2t6Orm_IgtLpk4xU4v64NYXXumO9d5DMKYdb5EJsWeKV1WoQZrHHuTFLpImH_gYij6ryp3Pkgbk wOJvaLkYEu3SDdMNX51.o.zzDtclP5Q9PHmvB8LbXGc4.ans6mA8qFNW.pHH0ycdG3FLy1O2Ko2c 6z658g_HIA8NcLJFKqFh9Mr_yDxRrVWquoVRwE8cVYmxgyyxSxAu6f12LuqHcgtrH2f5fhmV40fs F8YAKTjIrnNWpR6tW39dO_2UbZcz8oSsU2B1ktRNl90ZFcvLVw9JaJH0uUN7q1R2Sq.NQWtjImos wYPJ91wlbRQ8p.oZFcYJWBMew.igQbRaLgCJyn6rhFBLxqnVKtom6.75O2kQEaA7XjCb5EzLaJav VeEnliI2BSJV6b7vMdFFuzi0Y1j0EIEdfa3arDtDx9co.3A03VqxM2wlrXbQzlhShxAeQ4sCfliZ dJmFhplzoH.H2XTPnPfj0ikWruTUCORNcItzIMtS_HOodkUxF2NYBjW1jIepGI.mKVDd90_mm.SA ACg.yYG_jA_23h499nLlcWQXIINJ5BSJ2vL4Ms_xlR3nzrq3wHrSuiOq.9nuqjl.xNghCNbBP.DL gdDxq4XzYm_iDoi5lIWHgFQQVvPFE_7p3ie5mqCbwc5PaAQ_NNxuF3_dbBdNyHyQyJ_raD4BSfL4 2IBnteXnusfZK4GIXgDOvVNBi5FIhHitpFpLdzcG7GkOFQ9t3cOyOPEU7CGJdxBp9LtAOZ3i8Hkj ArZKMXQ.xoCs7h7_rpSrIX6LH4H7MQGPhHqrTpQnOJNxviFCmumu2TTGGhG_9YZrkpygg3WbmdFR irvB0mO7SI3uFkBh49C_vwHLKFLXYhN2tfI1H9EeS5yjZz8zVdF4v5KVY8P7yyoOgU7o1or8LEq3 U5iMtFUy2cNK7ISa8aOh4v_B8BR3fkxePwEkoN3yXR96hgEAd7tflyIYreDMxXcMOVyojRnTQBaV qUCs- X-Sonic-MF: X-Sonic-ID: b921d162-b21e-415b-adaa-41a2d40342f8 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Thu, 29 Feb 2024 16:06:52 +0000 Received: by hermes--production-gq1-5c57879fdf-hrd4s (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 3e2775b79f056de3cfc17447de35ab13; Thu, 29 Feb 2024 16:06:51 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task From: Mark Millard In-Reply-To: <426089C7-A15C-4B04-BC47-D1F77089C492@yahoo.com> Date: Thu, 29 Feb 2024 08:06:40 -0800 Cc: "Edward Sanford Sutton, III" Content-Transfer-Encoding: 7bit Message-Id: <46C1C5BA-924D-451E-8799-223F1A70413B@yahoo.com> References: <426089C7-A15C-4B04-BC47-D1F77089C492@yahoo.com> To: Peter , FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_CC(0.00)[hotmail.com]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[yahoo.com:+]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.147:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.147:from] X-Rspamd-Queue-Id: 4Tlx1V5Q5Pz4tgM [I grabbed locally modify text for one of those messages.] On Feb 29, 2024, at 08:02, Mark Millard wrote: > Peter 'PMc' Much wrote on > Date: Thu, 29 Feb 2024 13:40:05 UTC : > >> On 2024-02-27, Edward Sanford Sutton, III wrote: >>> More recently looked and see top showing threads+system processes >>> shows I have one core getting 100% cpu for kernel{arc_prune} which has >>> 21.2 hours over a 2 hour 23 minute uptime. >> >> Ack. >> >>> I started looking to see if >>> https://www.freebsd.org/security/advisories/FreeBSD-EN-23:18.openzfs.asc >>> was available as a fix for 13 but it is not (and doesn't quite sound >>> like it was supposed to apply to this issue). Would a kernel thread time >>> at 100% cpu for only 1 core explain the system becoming unusually >>> unresponsive? >> >> That depends. This arc_prune issue does usually go alongside with some >> other kernel thread (vm-whatever) also blocking, so you have two cores >> busy. How many remain? >> >> There is an updated patch in the PR 275594 (5 pieces), that works for >> 13.3; I have it installed, and only with that I am able to build gcc12 >> - otherwise the system would just OOM-crash (vm.pageout_oom_seq=5120 >> does not help with this). > > The kernel has multiple, distinct OOM messages. Which type are you > seeing? : > > "failed to reclaim memory" > "a thread waited too long to allocate a page" Local text: > "swblk or swpctrie zone exhausted" Should have been: "out of swap space" > "unknown OOM reason %d" > > Also, but only for boot verbose: > > "proc %d (%s) failed to alloc page on fault, starting OOM\n" > > > > vm.pageout_oom_seq is specific to delaying just: > "failed to reclaim memory" > === Mark Millard marklmi at yahoo.com From nobody Thu Feb 29 16:17:18 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlxFb1VMCz5CSf0 for ; Thu, 29 Feb 2024 16:17:23 +0000 (UTC) (envelope-from henrichhartzer@tuta.io) Received: from w1.tutanota.de (w1.tutanota.de [81.3.6.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.tutanota.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlxFZ3HmZz3xSy for ; Thu, 29 Feb 2024 16:17:22 +0000 (UTC) (envelope-from henrichhartzer@tuta.io) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuta.io header.s=s1 header.b=RLPMBsUq; dmarc=pass (policy=quarantine) header.from=tuta.io; spf=pass (mx1.freebsd.org: domain of henrichhartzer@tuta.io designates 81.3.6.162 as permitted sender) smtp.mailfrom=henrichhartzer@tuta.io Received: from tutadb.w10.tutanota.de (unknown [192.168.1.10]) by w1.tutanota.de (Postfix) with ESMTP id 3D5BEFBFC32 for ; Thu, 29 Feb 2024 16:17:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1709223438; s=s1; d=tuta.io; h=From:From:To:To:Subject:Subject:Content-Description:Content-ID:Content-Type:Content-Type:Content-Transfer-Encoding:Content-Transfer-Encoding:Cc:Date:Date:In-Reply-To:MIME-Version:MIME-Version:Message-ID:Message-ID:Reply-To:References:Sender; bh=jurMBshCdNmffrpobtKNRkJ7G1/iSL0N7DPZjw7aXbk=; b=RLPMBsUqJ4ZnWMnmB+w2WF4fR9yvvYitc9ANUFkd08nPliuxxVUkOn4jKYlUIyVs aXPFm2e/FZ7nzuNMVfmaNwHHUWOnLjEi8pD91wCo7+T2V4vIupxF9FqWiiPIWWnr61n U7iD21AZgaTd1E159rC+WdW62u74zd7dFlxKUdfr2yzT2KlC0oBYNfLbPVEkM+0VkpW X6wvrKm2CwAsRvCar4Qi8AVdHE9pCCeOTElWFs6fLak4qZiZhqw6m71oj1RfynaKeCu KbL3LA61yONEBnjnB1/h9DtZ2gAnnH+P53snFNgW8mkosO19Eb4A9ZiZN54pJxHrHl3 EKpJ0jjupw== Date: Thu, 29 Feb 2024 17:17:18 +0100 (CET) From: henrichhartzer@tuta.io To: Freebsd Stable Message-ID: Subject: service netif stop, instant reboot List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.10 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[tuta.io,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:81.3.6.160/28]; R_DKIM_ALLOW(-0.20)[tuta.io:s=s1]; RWL_MAILSPIKE_VERYGOOD(-0.20)[81.3.6.162:from]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; FROM_NO_DN(0.00)[]; ASN(0.00)[asn:24679, ipnet:81.3.0.0/18, country:DE]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[tuta.io:+] X-Rspamd-Queue-Id: 4TlxFZ3HmZz3xSy Hi, Not sure if this is the best list for this. I'm running an up-to-date 14.0-RELEASE. I ran `service netif stop` and my machine rebooted immediately. I haven't been able to find any logs or obvious errors. I tried reproducing the behavior and had no luck at all. I think it's the first time it's happened. This seems like something where the stars might have to align to cause the issue. This may be similar: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272607 May not be worth digging into unless someone else has seen similar behavior. Thanks! -Henrich From nobody Thu Feb 29 16:21:21 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlxTV4V7Bz5CTYR for ; Thu, 29 Feb 2024 16:27:42 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlxTV1L0kz41fw for ; Thu, 29 Feb 2024 16:27:42 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.18.1/8.18.1) with ESMTPS id 41TGR687083463 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 29 Feb 2024 17:27:07 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) ARC-Seal: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1709224029; cv=none; b=IMXKjiGpTDYbLDyhQq4y7QMfXET4TWQWVOQYmWLpBjDFT48JbjUXoRDug7FiTi+ODjC4TAZ1U/hUz9WR3rOtkjZWqIm55PInNYSMf/Vu4E72/C2jj3mnWtAJWClfohEn3ZdmWxrdD7c7rK9fDymOZBSejkRTDboFjQwoKjYDBLE= ARC-Message-Signature: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1709224029; c=relaxed/simple; bh=DkHTs+AH3PR6nesWeIMFhKNya8hNB72HshOByJZz4LU=; h=Received:Received:Received:Received:X-Authentication-Warning:Date: From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type: Content-Disposition:In-Reply-To:X-Milter:X-Greylist; b=aAj3xyQxL3qvryN8E5onejUjbxvAk0OJBG+Xzwh7KN29fqLoLSnAudru2yf12iJYB8IgExzfRzL43HOYlj423OhntByLj7vCMZJ9PDbg/86XWoHNoEBUE6tYh64T0/htcNevC1lXGNTVlWk47w5P5YAtNLliuKY+5a9OlXNqmwk= ARC-Authentication-Results: i=1; uucp.dinoex.org Received: (from uucp@localhost) by uucp.dinoex.org (8.18.1/8.18.1/Submit) with UUCP id 41TGR60V083457; Thu, 29 Feb 2024 17:27:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 41TGO61g094596 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Thu, 29 Feb 2024 17:24:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 41TGLLpR096836 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 29 Feb 2024 17:21:22 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 41TGLLlp096835; Thu, 29 Feb 2024 17:21:21 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Thu, 29 Feb 2024 17:21:21 +0100 From: Peter To: Mark Millard Cc: FreeBSD-STABLE Mailing List , "Edward Sanford Sutton, III" Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task Message-ID: References: <426089C7-A15C-4B04-BC47-D1F77089C492.ref@yahoo.com> <426089C7-A15C-4B04-BC47-D1F77089C492@yahoo.com> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <426089C7-A15C-4B04-BC47-D1F77089C492@yahoo.com> X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Thu, 29 Feb 2024 17:27:09 +0100 (CET) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Queue-Id: 4TlxTV1L0kz41fw On Thu, Feb 29, 2024 at 08:02:42AM -0800, Mark Millard wrote: ! Peter 'PMc' Much wrote on ! Date: Thu, 29 Feb 2024 13:40:05 UTC : ! ! > There is an updated patch in the PR 275594 (5 pieces), that works for ! > 13.3; I have it installed, and only with that I am able to build gcc12 ! > - otherwise the system would just OOM-crash (vm.pageout_oom_seq=5120 ! > does not help with this). ! ! The kernel has multiple, distinct OOM messages. Which type are you ! seeing? : ! ! "a thread waited too long to allocate a page" That one. From nobody Thu Feb 29 17:40:39 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tlz6441qVz5CbWr for ; Thu, 29 Feb 2024 17:41:00 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-54.consmr.mail.gq1.yahoo.com (sonic316-54.consmr.mail.gq1.yahoo.com [98.137.69.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tlz640htgz4BKv for ; Thu, 29 Feb 2024 17:40:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709228455; bh=G/WNWhygkk+WJzNFT9BzFe/4dffyo8Do3RqLrD02at4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=ExpBpZPkLZwgnIYp0DoraG4OGTSilNMsW0Wt9wA8NXUa6iDEbyinIzSWlYE1etxREA5H2Qp3/q66aJYs9nZCgRu7u1gw7bIiTvS3NjcyuOdmvhyLPgZKO0DrMt6j6GEwcgS+Om560J2PlJK+l4Cf+dZVIFwm/P+HpZXgiC7hZv+zo5Zw6wAjMAIq+fMTZCxKF2cHkd3bVSv8wRs3T3lEKEv+K47m2SzU+CgkLxVVCG1MYR/rrrdMVCW2aMmySn0MShdfdggY/nehNJ2Zfc06aWtPj7jhN7fapk0k6GZZ5NWMtk8vvY4WWKCpbRHWGrTPB+1cwEVDb0H9VcuTYnS4Mg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709228455; bh=QiQ0wj6Puuu14FzJh9CTwtcTEzAtRZFiNJI9YYpY4J3=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=cRGUyc8nO5zfTbPEG7/XN4Eeb245C5hJVhelf3WvocORk6TmPZisVn1rUint/+dee8CGnGro5dfcZuP4fOpaFI8ZHGjAbQ2Xe/qdXn/gB04hBwTclNSIRc+AgZx1LBbAnNkeRhOHA2IMx+rvbqcfK9UJgMjL76xbTHo8hC1EDdPu22F43XbFSX6/Om0V/CE7y8VTbVEhIH2PTLK0SPwYl8QgC0+9T93CGyivz1qFqyvUM23mAR5I5ZDyWuIDDSt6+V05V5Oz1LsLppgIa/uRQXdzRxNm5pYrTpWL6/wLJSVDX6+S8W/jmsQ3imz9L+S/A/epiKRYZeD7Z8LF0KXJjQ== X-YMail-OSG: tsCYWq4VM1nwah3FNzQSdSsZEIVniYjehoRd4E69tMD0teZOJ_oG6X8qr1M3S1M VLpAmeWLmpO6NfmqnaLfsjDhqoJhieBOpwpkuNTGqycpDFFaypjxLmzvi8NFTff2yml1Y.5LYO3H qOA1eOtsZlbFYQ8AHq9Fwx0_WIGVyAhqt2aE80F_ySk.MNnTXiMMXU_MDvdoX46UMQv.23.7eJbI BbpYnwIJwfe2YOt9xkvDd4w5tnG4THVAN1BUZIad.El4sw_7TAGuXZrOYXP_9JLUGPFaF8PThI3W 82yAfViLfZMzquh8pYg91omoI9UQEUnFl7w_LdgCLuIbnUnbNWNkYNn15m1hHOMWXYYq8dPkyDk2 9W3r_HLBTbL0Dj8T21bQFRswpT1fXV9_O3J7mPD6EYzrbnAD8lhLFr55pzHW2EruY7jMD18Jc.Fr hFZ30NjMgsg0jEyKYFOlopxfz1VKdGSmVVfH.oc4yNrFpBDgxbO_ERKOZdepED3lRRx1P9nPBZmA UBaKaX6.Q9YFefinVbFDA0fms8XW9Ri13pXzl858l_65MPvdvBNTpPnUXHPh7WZm8g._jEYXGDCk HOMxioIp0ZFyXCcwGiNe7t0W0.bltia.PPwGJoRYvEbl8jj66XFfXKSvadfIq7lS5gcsWVW4w0UL 8ouUtJ85aunH89Jwx5D06OJq6XMKF1noWf_1OxH4._6P4ciKte.veOLUIq2NHcBDO.HaOASn51if LzYcLJ8KaGU44o.gSyDs.3ebYUeIeE82GUEhJdDOUTgWreuh_bNcCqvs5AAYEPOBVLcl2_KAEiEm qQ_pAMmnp408amFW3i5q.lQS1EngG5gFGlrWidMAita7lap_2Pe0h2uiLvkp3kK25hDXXwKY.D1w cftFY40Cc0AAvHQm7uiJPJhBG0iu9Rv4MdgbtKNUzK2KCnKV9BVlK9rcM4pg9JMAI25W6vAqzg9V UObMTfvvSYzUxnLGzV2NtJ8mP.5NS6SEGKRbW4FnNfo14H5x3cAqdYB9D4HNQ9zpxBHjFL.gb63i ZmooZqpZ6h7S79b6PJla2Thry6zQXfSJz0p6bOVE4bq2N9FB7L0YIkU8HVj7Hd5oR67GsyvPu1kl UK_fjmz6DbM3dIz7W0PYYROTuMAVnaqAs1QLF7LNmAlBLL6NmuBtVxKX3wT6VPvh9.rPaEE4gqvU dUImgqXuq0CZAH4iGDY_tQ0Gjvyd.ylNNg2oCsSHpVjTzpgkNjPnvRgT0yeLpth1mi2zABSgWjU6 z6_QEtkdst5I0Ms9v2OprarG3IxXNwd0xtQPvCDj1QRpbL5fIi.5ih9ch3vO9Yg4l.uG.fKxVi3Z htxxIKUizpFgbRvnrBYYlBATJTjfOEef.uR7ydTF69CvLcQUD.ZBb0_86tcjsWIDpeY4A8pWHj6g HdUrwDQB69VMTfomTU35bWvE7ncIrmaoLjesglcevGg24Gz52QF4x4EXMVotBCRyn5L6gWRbzoMi 6fylPanwCW5qL8fF5bKNVTBttUwMZLhdDuUXqKDupfAJakuqEynx4iaI16ztdzjMf6kxC4kh0qpI Pk0e8uMWniQlRIW8BHYav9UZ_elnMB2fZ55457yxjpCcAGQLPOK5Kh1AzKF2iC14nroou0TlxUgG ib8B7xsCpeujtHZKsrHH8DTpziEV08f6nxflsruF.RF6CrDbCZHfnT4fX_CB4k.rI594RYe7UAwG SRHYYEMsiNpZd3k1vygFZbseYLR8O3.HPDYbG_oLosGJzUiRvF55hbribIH03HUyq8_rGj3nAkaf VqxRk29W0Ylw2rxzT4Xzzfb9nN3_Z5kWgIUWNFy2YKK5KnU3xomI0kAGOOT5R7SlZ6.ZT6ldzPQR nprfvssMd3PiMxz6jiXJnYHMLytfkLeajAkkP0V.PDeABtcuYlBHmvLHOo5iB2iGla7gUQpVAFYL OZ3wlR6QjET01R9Qb_o8UUMIzK1_97nK7CIjxi7HK6alHwRl5RmgVH4Qjpvtn2Yl_oi9P6lA.QBP Wmci2GJpNQoDr_Eoq5ZyqXsn8R54kmj444MBhfI3gY73e7LvaEBpmDlw8pvoN_UpQR1.76.LhxIa Xaqt.9O4N_G5iTZMwyDy0KuV4Z9U7UCWtIFOcnh9AqjhlBDA.DxanQB3FbcHYYgs8QpW6Sr92jOv rRjRmGN19qQymRFtF0zCnHegDAQQq5.I.L3fBZY7ZdChvFGR2BQsf6sqBoh7MsESTMht8QGoNDir xCP6CI50_cACozzXdZQBoiJXmlZXN9VqnkDPwYnnp6Obcz6bnaTaYVeN0iQRv3afElQD9swmp6w- - X-Sonic-MF: X-Sonic-ID: 810f35c7-91f2-441e-859e-e8d7f0c90700 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Thu, 29 Feb 2024 17:40:55 +0000 Received: by hermes--production-gq1-5c57879fdf-wt62k (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 70c0760451500192e9b5663e21249968; Thu, 29 Feb 2024 17:40:50 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task From: Mark Millard In-Reply-To: Date: Thu, 29 Feb 2024 09:40:39 -0800 Cc: FreeBSD-STABLE Mailing List , "Edward Sanford Sutton, III" Content-Transfer-Encoding: quoted-printable Message-Id: References: <426089C7-A15C-4B04-BC47-D1F77089C492.ref@yahoo.com> <426089C7-A15C-4B04-BC47-D1F77089C492@yahoo.com> To: Peter X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Queue-Id: 4Tlz640htgz4BKv On Feb 29, 2024, at 08:21, Peter wrote: > On Thu, Feb 29, 2024 at 08:02:42AM -0800, Mark Millard wrote: > ! Peter 'PMc' Much wrote on > ! Date: Thu, 29 Feb 2024 13:40:05 UTC : > !=20 > ! > There is an updated patch in the PR 275594 (5 pieces), that works = for > ! > 13.3; I have it installed, and only with that I am able to build = gcc12 > ! > - otherwise the system would just OOM-crash = (vm.pageout_oom_seq=3D5120 > ! > does not help with this). > !=20 > ! The kernel has multiple, distinct OOM messages. Which type are you > ! seeing? : > !=20 > ! "a thread waited too long to allocate a page" >=20 > That one. That explains why vm.pageout_oom_seq=3D5120 did not make a notable difference in the time frame. If you cause a verbose boot the code: if (bootverbose) printf( "proc %d (%s) failed to alloc page on fault, starting OOM\n", curproc->p_pid, curproc->p_comm); likely will report what process had failed to get a page in a timely manor. There also is control over the criteria for this but is is more complicated. In /boot/loader.conf (I'm using defaults): # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: #vm.pfault_oom_attempts=3D-1 # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes (showing defaults at the time): #vm.pfault_oom_attempts=3D 3 #vm.pfault_oom_wait=3D 10 # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) If you can be sure of not running out of swap/paging space, you might try vm.pfault_oom_attempts=3D-1 . If you do run out of swap/paging space, it would deadlock, as I understand. So, if you can tolerate that the -1 might be an option even if you do run out of swap/paging space. I do not have specific suggestions for alternatives to 3 and 10. It would be exploratory for me if I had to try such. For reference: # sysctl -Td vm.pfault_oom_attempts vm.pfault_oom_wait vm.pfault_oom_attempts: Number of page allocation attempts in page fault = handler before it triggers OOM handling vm.pfault_oom_wait: Number of seconds to wait for free pages before = retrying the page fault handler =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Feb 29 18:18:28 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tlzxk3jvqz5CfT7 for ; Thu, 29 Feb 2024 18:18:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tlzxj2JrTz4Ls4 for ; Thu, 29 Feb 2024 18:18:49 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=fLWRLuc6; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709230723; bh=VPuRNGepIq4tVClUGqwJTeptgGcvrmVIG6OcFfdewsI=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=fLWRLuc6N9y9WVvzPvGJ4zjV576gHk0mGoCOLV4yM4S8jlyurKpfJQXMAKEJfNWYKbhPWmiNRRQ/H8rtkQOSR/+1mDm4qrd0uSjk0eSfT5C5YpuAQZ9p4GsV/XDgTBXpl2d10tMSxhxoOEkKupUA3vYX2uddl8KpLqy7aTzfd0Jbe8z5+LARuXAepa+BMPkgd8b4hm8DnTtat6Cw5KQrjWkkQrGj5/ho3Dez+dBv8JBdVpWXVoB1fO+FY0PLv8hJ2PWLyepBu17KDXdI6RQtfISDwaGPwBpLKfq986a+omgNTuD/8IfFTIZ6eJrEzxFEn6/dO+JnSgRN2ANXQtBsNg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709230723; bh=Xyx7vWLdtUUzCTXsDqwopM6GAHpZv/9tqihGo2z3rF4=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=Nu9V2ieA7L1s5RTCcnBHVSvgPYh5bQR09IAXB8PPamkkXO1rVVgpFqaG0Pjcd8jUV7U84Oou1HmBnAlxJsGPXicI2SE45gOuw3jsMQ558XZ3H8ecnFdLJ7lsse7/8KOK0KY+6gPtbkGkgoDj05241hyIO90k896O+9sJuPrzWSE60dkGbjmMyFAQck8VmvdU8K7X/xcW4wwCe2FcrEM9UaVzdKwjpLPiBNpJ8w+EdcSVIfUQTpkloz4kFEr/TMvabZcn4Fc/aIqe9GuXqUcLDnwBr4YS+MhScR7axJK31vl5l8egn8bhyhGo6vKSHSSlzkjJZBymIHv1RdGvp3M62Q== X-YMail-OSG: j7SlnusVM1lOfNBAIYpnByq.irZHGfrmGvQZV.iL1PFNm6ALy824RNYC8XIFIJ3 hL4DsjhICO1Uko7KYDAtVVFObYE9eVSIU6sD.Dn6BuggkhROma63NgSJ9Zy0i926nW3qMzdrEGua 0ybef0v70drNP.id68NDuOd7SPF24dZ_yEXss3I9M6AAaRmcAjTwiBP.15Hz_5LEQEKQsYOpYnr2 n9z9tTpA6wHmt5EiYyGn_sa74s3N.Jm8fM6pTUMsO1wq0j8JnWCK9vPsl7z62TWSUheX7hWsFmcR VOb8eHY5ZokXhHrjBpbloUnPF3OW0oYKujbkKW1iUEUdVbSdjX.IIqdUl166oMy5ylDUZtJbrhoA knYmXbDDg.cKehaGxRFeinLQSuLYHGpW0a4SX.0pRfrR5wP4hcdOm4bsmBxroszu6yM6_e4Ozmkm 6vPvfsMTNQM5XPG.83TwBO1e4XvjJR.sGD6OXZRO_UYnh9DZhCAUNLKxr8NT.AP4f0iJTBXDmp0W aOaImxrMNoVl_1C9Hd..tBZJbkEdZ_II9T6EGLEpgvJrIfuhIWjYKBvN9ge6xLdNF0j2Hz7N8VbQ FnwsS5T7wyPRIPPeuptut1OUO8Clp7MMPMpbYT4UP6pyrVZswTZJA9FtptO7R3AJRml.4sU5UeCe PwjJclOpIg_ZOJtu3TRr1kxbZ67QpKtEAKWhsh4ZAO6n2iEAfGquwdTLmQykkYz2kDXSTnhiHqNu _C0jJacQlfVZqmA3xTa7hUJ4TBe4Myim3OGsQmGSpn_xZh_G9qUcDQqWS4USgXM50a7DXLMARTX5 UIfR.ZIGlG5MmVyXkZeVdhx0z.sMmInePhktQwYxpQ1dj9z5oqy62axKfcME3w7WCW5CBxnt60zo Gy.DnLxS0fA4_9QtJ8Mm85fMn.rS_wJIsY166XtoZAMGaYasLQKZzxAvMyI3w_GgYV7MVh.gc_VB lMUNu5lREqtOkFQk8aSZ5qJSviS6PZvXboCw7jPgSCCkhB414lWjZEis3hnOgmqBwCT9ZCamI19G 8awB6pqzSwf8JaSKamtwgTfPskaJxWo6jblafz2j1DUtIypQ0hX4sEqHe5N.Hol2BN0XvZS5.fOX THuiz73r5kvZFSmkftCmTbApL59oIhH3CNIH696COR1CjDWxi3.dQ.Ymkn1M4YwRgaMo84y6Sj_A 5hJMfiYoBwNyeBcv0aiXDRFcNLqYBrfRWJV9xQR9XBUA0kVlDlIKs0gRqmsRQnv2Bw1Fiz87LpNo 5mF9OVv8O2qY_ZAxMW27lh1Z9Zku0kgXnEzOR6dtavadyO38HyPawgUZxsIzC38KmBsQSfT6RV1K 0BPRsRIFMY50_XSw_iW0PEoTBeHoMJRBv_kstk7WKz.uPIPFntVkflMCMXO1_CnlLGjn4hZi9buk yuGkdBMrw01_H3RA6EJRqgZd6CxwgzR0nxmsUZTO3Mz6iGWL0Fk725iOab6M7gR_rx4Zi.ugJRL. tzM2ba71x2vnLxCSUB2Klpgp_9tLjOeytJ4K7rRFReG96DGNcDbrbyrYWCaHcJId5s9Ji97eBrJw t.EbX14W5MzSiC0.x8rZB6N_.jpwtcx7frKbjsePZPkMTBrw4qrVcwpH_oYCRT4HgUMxEKJGnaLV ofuA3kDJQyAkmyJGi9ZT1e.tJw5oKXAycTdxQtbiysTYibsJUxaTGib3h2a0L4Ok1DB5cEMOKM.6 9QJtGU8cx1FMVhwmAfpHK9Bq4MxiqFB2PXigTucq9MxmE7jySAvvM9xlt9VlFKOqmfQ5rZN73._I TToiKClS7XQnGVsIkR1KRY1dxnZhnmI94fJqTXtNNz_7vbaHOeEgdHLdwSJDUGs2wzzVfKDuus41 Tv9g.9rX3XVcczhFqFHq3Ogsg4xvuFqFgLjMJHySXTBN5a81A11AKibyUseMOkBh2SqtNr64hEAO 3xyaYEA.ElkwULDV4DzDrW3GYq1BTba80XFOtK8vGSrfy1N0J.Nu4WcPOUJnH3V3SQYESMLQbcKd 3C5k.v3j51kkVtt1qVEDb2p5qqb26GqklzQTVXFObGz6QmY8NuhsVgJQrv9BWtfxTy1e2RMZjMec Tcmv_hPFRVO8hqCKQ3d3._d0IzxEceYN606u7GxVJ_uIYeUaS.WrHjrVCbOXzJavfURBummuxoZf UpB_FBahOCW4hlkQF2.O0WQAbX.cpyFoqdy.vzRhVUlUULbtUpDCGWcUcF9NrdH.PWpq11roALR7 qLIaYLVGnzl8OaB_lBmVnZXmNKF0PcDUSXG3z0DFkUZOkgLSFKG063c.6LIE_5x6Uxv3KYoiEGMN wEQ-- X-Sonic-MF: X-Sonic-ID: d2a30b31-2869-4527-8437-6d0f46e294e8 Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Thu, 29 Feb 2024 18:18:43 +0000 Received: by hermes--production-gq1-5c57879fdf-hjdnf (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID a5961f765e20f00975aa8b08edb4cbb7; Thu, 29 Feb 2024 18:18:39 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task From: Mark Millard In-Reply-To: Date: Thu, 29 Feb 2024 10:18:28 -0800 Cc: FreeBSD-STABLE Mailing List , "Edward Sanford Sutton, III" Content-Transfer-Encoding: quoted-printable Message-Id: <79745A1B-0061-40FB-89C3-E71893D0D18D@yahoo.com> References: <426089C7-A15C-4B04-BC47-D1F77089C492.ref@yahoo.com> <426089C7-A15C-4B04-BC47-D1F77089C492@yahoo.com> To: Peter X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_CC(0.00)[freebsd.org,hotmail.com]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[yahoo.com:+]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.205:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.205:from] X-Rspamd-Queue-Id: 4Tlzxj2JrTz4Ls4 On Feb 29, 2024, at 09:40, Mark Millard wrote: > On Feb 29, 2024, at 08:21, Peter wrote: >=20 >> On Thu, Feb 29, 2024 at 08:02:42AM -0800, Mark Millard wrote: >> ! Peter 'PMc' Much wrote on >> ! Date: Thu, 29 Feb 2024 13:40:05 UTC : >> !=20 >> ! > There is an updated patch in the PR 275594 (5 pieces), that works = for >> ! > 13.3; I have it installed, and only with that I am able to build = gcc12 >> ! > - otherwise the system would just OOM-crash = (vm.pageout_oom_seq=3D5120 >> ! > does not help with this). >> !=20 >> ! The kernel has multiple, distinct OOM messages. Which type are you >> ! seeing? : >> !=20 >> ! "a thread waited too long to allocate a page" >>=20 >> That one. >=20 > That explains why vm.pageout_oom_seq=3D5120 did not make a > notable difference in the time frame. >=20 > If you cause a verbose boot the code: >=20 > if (bootverbose) > printf( > "proc %d (%s) failed to alloc page on fault, starting = OOM\n", > curproc->p_pid, curproc->p_comm); >=20 > likely will report what process had failed to get a > page in a timely manor. >=20 > There also is control over the criteria for this but is > is more complicated. In /boot/loader.conf (I'm using > defaults): >=20 > # > # For plunty of swap/paging space (will not > # run out), avoid pageout delays leading to > # Out Of Memory killing of processes: > #vm.pfault_oom_attempts=3D-1 > # > # For possibly insufficient swap/paging space > # (might run out), increase the pageout delay > # that leads to Out Of Memory killing of > # processes (showing defaults at the time): > #vm.pfault_oom_attempts=3D 3 > #vm.pfault_oom_wait=3D 10 > # (The multiplication is the total but there > # are other potential tradoffs in the factors > # multiplied, even for nearly the same total.) >=20 > If you can be sure of not running out of swap/paging > space, you might try vm.pfault_oom_attempts=3D-1 . > If you do run out of swap/paging space, it would > deadlock, as I understand. So, if you can tolerate > that the -1 might be an option even if you do run > out of swap/paging space. >=20 > I do not have specific suggestions for alternatives > to 3 and 10. It would be exploratory for me if I had > to try such. >=20 > For reference: >=20 > # sysctl -Td vm.pfault_oom_attempts vm.pfault_oom_wait > vm.pfault_oom_attempts: Number of page allocation attempts in page = fault handler before it triggers OOM handling > vm.pfault_oom_wait: Number of seconds to wait for free pages before = retrying the page fault handler I'll note that vm.pageout_oom_seq , vm.pfault_oom_attempts , and vm.pfault_oom_wait are all live writable, not just boot-time tunables. In other words, all show a line of output in: # sysctl -Wd vm.pageout_oom_seq vm.pfault_oom_attempts = vm.pfault_oom_wait vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM vm.pfault_oom_attempts: Number of page allocation attempts in page fault = handler before it triggers OOM handling vm.pfault_oom_wait: Number of seconds to wait for free pages before = retrying the page fault handler Not just in: # sysctl -Td vm.pageout_oom_seq vm.pfault_oom_attempts = vm.pfault_oom_wait vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM vm.pfault_oom_attempts: Number of page allocation attempts in page fault = handler before it triggers OOM handling vm.pfault_oom_wait: Number of seconds to wait for free pages before = retrying the page fault handler (To see values, to not use the "d".) =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Feb 29 18:45:04 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tm0XL74s8z5Chqf for ; Thu, 29 Feb 2024 18:45:22 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Received: from udns.ultimatedns.net (udns.ultimatedns.net [24.113.41.81]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "ultimatedns.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tm0XL43L5z4Pbf for ; Thu, 29 Feb 2024 18:45:22 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Authentication-Results: mx1.freebsd.org; none Received: from ultimatedns.net (localhost [127.0.0.1]) by udns.ultimatedns.net (8.16.1/8.16.1) with ESMTP id 41TIj46Q053956; Thu, 29 Feb 2024 10:45:16 -0800 (PST) (envelope-from bsd-lists@bsdforge.com) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Date: Thu, 29 Feb 2024 10:45:04 -0800 From: Chris To: henrichhartzer@tuta.io Cc: Freebsd Stable Subject: Re: service netif stop, instant reboot In-Reply-To: References: User-Agent: UDNSMS/17.0 Message-ID: <126e5c0e8f699ef2965d3ed4ac2b007e@bsdforge.com> X-Sender: bsd-lists@bsdforge.com Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11404, ipnet:24.113.0.0/16, country:US] X-Rspamd-Queue-Id: 4Tm0XL43L5z4Pbf On 2024-02-29 08:17, henrichhartzer@tuta.io wrote: > Hi, > > Not sure if this is the best list for this. > > I'm running an up-to-date 14.0-RELEASE. I ran `service netif stop` and my > machine > rebooted immediately. I haven't been able to find any logs or obvious > errors. > > I tried reproducing the behavior and had no luck at all. I think it's the > first > time it's happened. > > This seems like something where the stars might have to align to cause the > issue. > > This may be similar: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272607 > > May not be worth digging into unless someone else has seen similar behavior. > > Thanks! > > -Henrich I ran into a case very similar with a `service netif restart` Which was later addressed in at least work done on iwlwifi(4). In an effort for anyone to better serve you. A summary of what your on and what hardware you're running it on would be necessary. :) --Chris From nobody Thu Feb 29 18:49:08 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tm0cv2fWzz5CjGf for ; Thu, 29 Feb 2024 18:49:19 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from fc.opsec.eu (fc.opsec.eu [IPv6:2001:14f8:200:4::4]) by mx1.freebsd.org (Postfix) with ESMTP id 4Tm0ct4xBfz4Qhd for ; Thu, 29 Feb 2024 18:49:18 +0000 (UTC) (envelope-from pi@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from pi (uid 104) (envelope-from pi@freebsd.org) id 2ccd6 by fc.opsec.eu (DragonFly Mail Agent v0.13+ on fc.opsec.eu); Thu, 29 Feb 2024 19:49:08 +0100 Date: Thu, 29 Feb 2024 19:49:08 +0100 From: Kurt Jaeger To: henrichhartzer@tuta.io Cc: Freebsd Stable Subject: Re: service netif stop, instant reboot Message-ID: References: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:12502, ipnet:2001:14f8::/32, country:DE] X-Rspamd-Queue-Id: 4Tm0ct4xBfz4Qhd Hi! > I tried reproducing the behavior and had no luck at all. > I think it's the first time it's happened. Did it happen with a running WLAN ? I have similar issues if WLAN is active. > This seems like something where the stars might have to align to cause the issue. > > This may be similar: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272607 Indeed, interesting! -- pi@FreeBSD.org +49 171 3101372 Now what ? From nobody Thu Feb 29 18:56:10 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tm0sY25bDz5Cjsc for ; Thu, 29 Feb 2024 19:00:17 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tm0sX4nYFz4Svb for ; Thu, 29 Feb 2024 19:00:16 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.18.1/8.18.1) with ESMTPS id 41TJ05pf095031 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 29 Feb 2024 20:00:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) ARC-Seal: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1709233208; cv=none; b=KsXwp8iksU8L4fbWe45lZke70fe1pRUJmDLM2isoN+IKt4Ko5QbEFdRM3Hs2jLpCeY3OBFtwSDvfb236aocSpQib82drcE/Kb8E4mHoeEotsc5Gobbkf8nbF9jrEn7uhq9YCq+Qlji+jkKjrGn1ZkCegdyjPcIVcsHcxeMvx8P4= ARC-Message-Signature: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1709233208; c=relaxed/simple; bh=gIlLGLreNtVVenpmmR58Vowu1nQDVej0e77DSTRO0DU=; h=Received:Received:Received:Received:X-Authentication-Warning:Date: From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type: Content-Disposition:In-Reply-To:X-Milter:X-Greylist; b=HdnHjk1Z9kDVy6WRvpDZ1iVGzi4EwOAp4UVvrl0uNcd5kcrxYGTfXbl5BH8yOJVX0ps6EzCGt3YRuZqUpfMZy4TyJ3rEKafWYJPbPnfTPo1gZPeYxeZ8EJvrsGYtHTcmRA2mCheTS0YTHGWq1nKee4LVXcDK71/3euemaLqcr6U= ARC-Authentication-Results: i=1; uucp.dinoex.org Received: (from uucp@localhost) by uucp.dinoex.org (8.18.1/8.18.1/Submit) with UUCP id 41TJ05Gj095030; Thu, 29 Feb 2024 20:00:05 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 41TIv6CB071627 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Thu, 29 Feb 2024 19:57:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 41TIuAdO097704 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 29 Feb 2024 19:56:12 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 41TIuAZ6097703; Thu, 29 Feb 2024 19:56:10 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Thu, 29 Feb 2024 19:56:10 +0100 From: Peter To: Mark Millard Cc: FreeBSD-STABLE Mailing List , "Edward Sanford Sutton, III" Subject: Re: 13-STABLE high idprio load gives poor responsiveness and excessive CPU time per task Message-ID: References: <426089C7-A15C-4B04-BC47-D1F77089C492.ref@yahoo.com> <426089C7-A15C-4B04-BC47-D1F77089C492@yahoo.com> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Thu, 29 Feb 2024 20:00:08 +0100 (CET) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Queue-Id: 4Tm0sX4nYFz4Svb First off: the honorable Sir Edward, III. had the decency to have me ntified that they prefer to censor my messages (reasons were not given), I for my part consider it rather pointless to publicly ask questions, only to then inform those who bother to answer that one declines to receive their answers. So much for that. Now for the call for popcorn: On Thu, Feb 29, 2024 at 09:40:39AM -0800, Mark Millard wrote: ! > ! The kernel has multiple, distinct OOM messages. Which type are you ! > ! seeing? : ! > ! ! > ! "a thread waited too long to allocate a page" ! > ! > That one. ! ! That explains why vm.pageout_oom_seq=5120 did not make a ! notable difference in the time frame. Good. Glad it explains something. ! If you cause a verbose boot the code: ! ! if (bootverbose) ! printf( ! "proc %d (%s) failed to alloc page on fault, starting OOM\n", ! curproc->p_pid, curproc->p_comm); ! ! likely will report what process had failed to get a ! page in a timely manor. These are ad-hoc bhyve which are only created for the purpose of compiling some ports. So there is zero interest about /which/ process fails, because any failing process will just fail the build. The essential point it rather: the very same sizing of ressources works when booting a 13.2, and crashes reproducible with 13.3 ! # run out), avoid pageout delays leading to ! # Out Of Memory killing of processes: ! #vm.pfault_oom_attempts=-1 Yes, I already got that far, and that doesn't help: If the system is neither allowed to oom-kill nor to crash, it freezes and waits for the reset button. As this is an endless loop in the kernel, it is not ressource exhaustion, but rather unability of the kernel to adjust ressources accordingly due to being busy with other things (i.e. running an endless loop). But then, discussion about this is futile, because there exists a patch that well expects and nicely fixes mentioned behaviour. From nobody Thu Feb 29 23:30:42 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tm6sx6CZwz5D6Xf for ; Thu, 29 Feb 2024 23:31:01 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tm6sx0cmGz42KW; Thu, 29 Feb 2024 23:31:01 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=ipZ54TSQ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::635 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1dc0e5b223eso14632465ad.1; Thu, 29 Feb 2024 15:31:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709249459; x=1709854259; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=agjotOpTK8y1NLx7JS/G4SuRbAX2kP7IXS8jsN3catY=; b=ipZ54TSQN7kGrVYhbU2bfOcupfgLCG5sc/eYwtZTtIMkQXhgDPtQ/x0J8j28FeLw+e TwwjDK0gn1UF6W5L5wzAeo25gI+dUUy34gOM0FkPE6oIvs7/WjsUTHC0nck78+bCAWSH nlMukrdccWNbwJVhSXABCa3cP0MNAQeJLYxDO+zyWsk9j0k9QcWbqxrLOtgthrn9gdxu XtGEXQ0OyKQyb2M7iA5DqZI8B9KAhgZhrOqlSqonMCjeX47HPFS2IW0Nl+JeXuFnyK6a RAXHJunYptQ2zOkANNRDP96sQhs4FOVmBSxHrdSA3NFDfZE2fek+y/OhMBcmYq+y7DL/ xIkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709249459; x=1709854259; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=agjotOpTK8y1NLx7JS/G4SuRbAX2kP7IXS8jsN3catY=; b=gM9ze7spLqaz6A4Z1IaDF3UuYu6ZO9b4p2XLqL8XuUYhOkxmughfTUPR6pbOgGjfiw 87y0OoHzDiMQ+b7paoSt/LgGD1ftMkF9jh72XFmQ59C3I59QB1ZrKns8zdtyW3gWbbVg n003U2Jp9zvJ+idBwnCwAu+a7WeG1x2AcPhPd9DpsKE7Un7N96trQQv/OfMtMdseCHl8 u9TIByN+XbskNorsmpbOnd6/VUoTY3QnHv5YspPSwYKXeG2u0AcbXpP3JgmyzQkuMx1B CqIM5Z5FQoriDdJTlt7P049+abtF3aqFknmMlk3CbP5icGrO5Yz08cwSm84xZyLCo2lH OaKA== X-Forwarded-Encrypted: i=1; AJvYcCUC5y0ulqcrTreUhBqhlb06Pc0ZlpqH1Q2PEEZLQvNF2ywhQBs0SyoSS33gyFKQ+WByqQkZxWZ7/QYJDkQYs/CH4SLxPw== X-Gm-Message-State: AOJu0YxqcMy5ZjYcm4ICT2wZvXuktKqXTZQYh1hzqB3OXkgQIGU7w+CK yG/dVPVUKSSooupLUEAc+kelLsherQpnFc03N5Cg1HDjdv9lHoxnqA2LNAWPNZrEwBCfQUDayPZ 3OFuqAbbQO+dZZMa6bzr5X/ZvGDxlVwA= X-Google-Smtp-Source: AGHT+IFABc39YSKGulEHrxAILgMbWNKFRTC8TCHc9FD8gKvbW6xFppD2PnADwruUPyEfjTPXGi9AFND7S28WA1rMWSM= X-Received: by 2002:a17:902:f552:b0:1dc:7b6:867a with SMTP id h18-20020a170902f55200b001dc07b6867amr24685plf.21.1709249459040; Thu, 29 Feb 2024 15:30:59 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <26078.50375.679881.64018@hergotha.csail.mit.edu> In-Reply-To: From: Rick Macklem Date: Thu, 29 Feb 2024 15:30:42 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Garrett Wollman Cc: stable@freebsd.org, rmacklem@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.97 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.971]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_FROM(0.00)[]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::635:from]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4Tm6sx0cmGz42KW On Wed, Feb 28, 2024 at 4:04=E2=80=AFPM Rick Macklem wrote: > > On Tue, Feb 27, 2024 at 9:30=E2=80=AFPM Garrett Wollman wrote: > > > > Hi, all, > > > > We've had some complaints of NFS hanging at unpredictable intervals. > > Our NFS servers are running a 13-stable from last December, and > > tonight I sat in front of the monitor watching `nfsstat -dW`. I was > > able to clearly see that there were periods when NFS activity would > > drop *instantly* from 30,000 ops/s to flat zero, which would last > > for about 25 seconds before resuming exactly as it was before. > > > > I wrote a little awk script to watch for this happening and run > > `procstat -k` on the nfsd process, and I saw that all but two of the > > service threads were idle. The three nfsd threads that had non-idle > > kstacks were: > > > > PID TID COMM TDNAME KSTACK > > 997 108481 nfsd nfsd: master mi_switch sleepq_t= imedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc= _run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > > 997 960918 nfsd nfsd: service mi_switch sleepq_t= imedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc n= fssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > 997 962232 nfsd nfsd: service mi_switch _cv_wait= txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd= _ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANG= E vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc= _run_internal svc_thread_start fork_exit fork_trampoline > > > > I'm suspicious of two things: first, the copy_file_range RPC; second, > > the "master" nfsd thread is actually servicing an RPC which requires > > obtaining a lock. The "master" getting stuck while performing client > > RPCs is, I believe, the reason NFS service grinds to a halt when a > > client tries to write into a near-full filesystem, so this problem > > would be more evidence that the dispatching function should not be > > mixed with actual operations. I don't know what the clients are > > doing, but is it possible that nfsrvd_copy_file_range is holding a > > lock that is needed by one or both of the other two threads? > > > > Near-term I could change nfsrvd_copy_file_range to just > > unconditionally return NFSERR_NOTSUP and force the clients to fall > > back, but I figured I would ask if anyone else has seen this. > I have attached a little patch that should limit the server's Copy size > to vfs.nfsd.maxcopyrange (default of 10Mbytes). > Hopefully this makes sure that the Copy does not take too long. > > You could try this instead of disabling Copy. It would be nice to know if > this is suffciient? (If not, I'll probably add a sysctl to disable Copy.) I did a quick test without/with this patch,where I copied a 1Gbyte file. Without this patch, the Copy RPCs mostly replied in just under 1sec (which is what the flag requests), but took over 4sec for one of the Copy operations. This implies that one Read/Write of 1Mbyte on the server took over 3 seconds. I noticed the first Copy did over 600Mbytes, but the rest did about 100Mbyt= es each and it was one of these 100Mbyte Copy operations that took over 4sec. With the patch, there were a lot more Copy RPCs (as expected) of 10Mbytes each and they took a consistent 0.25-0.3sec to reply. (This is a test of a = local mount on an old laptop, so nowhere near a server hardware config.) So, the patch might be sufficient? It would be nice to avoid disabling Copy, since it avoids reading the data into the client and then writing it back to the server. I will probably commit both patches (10Mbyte clip of Copy size and disabling Copy) to main soon, since I cannot say if clipping the size of the Copy will always be sufficient. Pleas let us know how trying these patches goes, rick > > rick > > > > > -GAWollman > > > > From nobody Fri Mar 1 08:00:20 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TmL9q1mcHz5C8NV for ; Fri, 1 Mar 2024 08:00:31 +0000 (UTC) (envelope-from SRS0=zTW4=KH=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int-backup.realworks.nl (smtp-relay-int-backup.realworks.nl [87.255.56.188]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TmL9n4q1Tz4pb9; Fri, 1 Mar 2024 08:00:29 +0000 (UTC) (envelope-from SRS0=zTW4=KH=klop.ws=ronald-lists@realworks.nl) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=rw2 header.b=pSRAALN+; dmarc=pass (policy=quarantine) header.from=klop.ws; spf=pass (mx1.freebsd.org: domain of "SRS0=zTW4=KH=klop.ws=ronald-lists@realworks.nl" designates 87.255.56.188 as permitted sender) smtp.mailfrom="SRS0=zTW4=KH=klop.ws=ronald-lists@realworks.nl" Received: from smtp-relay-int.realworks.nl (rwvirtual375.colo.realworks.nl [10.0.10.75]) by mailrelayint1.colo2.realworks.nl (Postfix) with ESMTP id 4TmL9d2fctz1Zg; Fri, 1 Mar 2024 09:00:21 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1709280021; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=6YLdcB4oAa81IK4ne6Yd37g++Gdb9f2xIaikl8IJM8I=; b=pSRAALN+rS44Dh+idXsYqS1/5g1jKlHIYwThYs8d5Rp1DHFUgD8SisEXwlTZZHNNVgv7XS 7GKQN+XWhT7ZF3drNO9gWE9NqitWlHz0zRNy4j1SoZZ1di+HD0WYZCdIGF/FbRntQJ/lbo 4WmMCCTz2JmaughjVJsVQ5d/k6Ui4PbB9NzerEHQKJ3yElMcEBUHCOZhLAAopHsOKbjAys 3kKRklhn8smBwCg7GtlKXiYMNmys1ZvvLTlhta/rui6mm97V/z9/dk9XMJQzLlSFLqjx8I rviIFsl3Phe0V6FxjJJCZ0oXLABgTK8nGSJ1YvuwJ9NLDVpbhVLBBL0xw1dT8Q== Received: from rwvirtual375.colo.realworks.nl (localhost [127.0.0.1]) by rwvirtual375.colo.realworks.nl (Postfix) with ESMTP id 036DD401A9; Fri, 1 Mar 2024 09:00:20 +0100 (CET) Date: Fri, 1 Mar 2024 09:00:20 +0100 (CET) From: Ronald Klop To: Rick Macklem Cc: Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Message-ID: <1020651467.1592.1709280020993@localhost> In-Reply-To: Subject: Re: 13-stable NFS server hang List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1591_1207250147.1709280020983" X-Mailer: Realworks (692.38) X-Originating-Host: from (localhost [127.0.0.1]) by rwvirtual375 [10.0.10.75] with HTTP; Fri, 01 Mar 2024 09:00:20 +0100 Importance: Normal X-Priority: 3 (Normal) X-Spamd-Bar: - X-Spamd-Result: default: False [-1.70 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,quarantine]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[ronald-lists@klop.ws,SRS0=zTW4=KH=klop.ws=ronald-lists@realworks.nl]; R_SPF_ALLOW(-0.20)[+ip4:87.255.56.128/26]; R_DKIM_ALLOW(-0.20)[klop.ws:s=rw2]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; ASN(0.00)[asn:38930, ipnet:87.255.32.0/19, country:NL]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; HAS_X_PRIO_THREE(0.00)[3]; TAGGED_RCPT(0.00)[]; FROM_NEQ_ENVFROM(0.00)[ronald-lists@klop.ws,SRS0=zTW4=KH=klop.ws=ronald-lists@realworks.nl]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; RCPT_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[klop.ws:+] X-Rspamd-Queue-Id: 4TmL9n4q1Tz4pb9 ------=_Part_1591_1207250147.1709280020983 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Interesting read. Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations? Like ongoing file operations could have a read only view/copy of the mount table. Only new operations will have to wait. But the mount never needs to wait for ongoing operations before locking the structure. Just a thought in the morning Regards, Ronald. Van: Rick Macklem Datum: 1 maart 2024 00:31 Aan: Garrett Wollman CC: stable@freebsd.org, rmacklem@freebsd.org Onderwerp: Re: 13-stable NFS server hang > > > On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote: > > > > On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote: > > > > > > Hi, all, > > > > > > We've had some complaints of NFS hanging at unpredictable intervals. > > > Our NFS servers are running a 13-stable from last December, and > > > tonight I sat in front of the monitor watching `nfsstat -dW`. I was > > > able to clearly see that there were periods when NFS activity would > > > drop *instantly* from 30,000 ops/s to flat zero, which would last > > > for about 25 seconds before resuming exactly as it was before. > > > > > > I wrote a little awk script to watch for this happening and run > > > `procstat -k` on the nfsd process, and I saw that all but two of the > > > service threads were idle. The three nfsd threads that had non-idle > > > kstacks were: > > > > > > PID TID COMM TDNAME KSTACK > > > 997 108481 nfsd nfsd: master mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > > > 997 960918 nfsd nfsd: service mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > > 997 962232 nfsd nfsd: service mi_switch _cv_wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > I'm suspicious of two things: first, the copy_file_range RPC; second, > > > the "master" nfsd thread is actually servicing an RPC which requires > > > obtaining a lock. The "master" getting stuck while performing client > > > RPCs is, I believe, the reason NFS service grinds to a halt when a > > > client tries to write into a near-full filesystem, so this problem > > > would be more evidence that the dispatching function should not be > > > mixed with actual operations. I don't know what the clients are > > > doing, but is it possible that nfsrvd_copy_file_range is holding a > > > lock that is needed by one or both of the other two threads? > > > > > > Near-term I could change nfsrvd_copy_file_range to just > > > unconditionally return NFSERR_NOTSUP and force the clients to fall > > > back, but I figured I would ask if anyone else has seen this. > > I have attached a little patch that should limit the server's Copy size > > to vfs.nfsd.maxcopyrange (default of 10Mbytes). > > Hopefully this makes sure that the Copy does not take too long. > > > > You could try this instead of disabling Copy. It would be nice to know if > > this is suffciient? (If not, I'll probably add a sysctl to disable Copy.) > I did a quick test without/with this patch,where I copied a 1Gbyte file. > > Without this patch, the Copy RPCs mostly replied in just under 1sec > (which is what the flag requests), but took over 4sec for one of the Copy > operations. This implies that one Read/Write of 1Mbyte on the server > took over 3 seconds. > I noticed the first Copy did over 600Mbytes, but the rest did about 100Mbytes > each and it was one of these 100Mbyte Copy operations that took over 4sec. > > With the patch, there were a lot more Copy RPCs (as expected) of 10Mbytes > each and they took a consistent 0.25-0.3sec to reply. (This is a test of a local > mount on an old laptop, so nowhere near a server hardware config.) > > So, the patch might be sufficient? > > It would be nice to avoid disabling Copy, since it avoids reading the data > into the client and then writing it back to the server. > > I will probably commit both patches (10Mbyte clip of Copy size and > disabling Copy) to main soon, since I cannot say if clipping the size > of the Copy will always be sufficient. > > Pleas let us know how trying these patches goes, rick > > > > > rick > > > > > > > > -GAWollman > > > > > > > > > > > ------=_Part_1591_1207250147.1709280020983 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: quoted-printable Interesting read. 

 W= ould it be possible to separate locking for admin actions like a client mou= nting an fs from traffic flowing for file operations?

<= div>Like ongoing file operations could have a read only view/copy of the mo= unt table. Only new operations will have to wait.
But the mount n= ever needs to wait for ongoing operations before locking the structure.&nbs= p;

Just a thought in the morning

Regards,
Ronald.

Van: Rick Macklem <rick.macklem@gmail.com>
Datum: 1 maart 2024 00:31
Aan: Garrett Wollman <wollman@b= imajority.org>
CC: stable@freebsd.org, rmacklem@free= bsd.org
Onderwerp: Re: 13-stable NFS server hang

On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote:
>
> On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote:
> >
> > Hi, all,
> >
> > We've had some complaints of NFS hanging at unpredictable interva= ls.
> > Our NFS servers are running a 13-stable from last December, and > > tonight I sat in front of the monitor watching `nfsstat -dW`. &nb= sp;I was
> > able to clearly see that there were periods when NFS activity wou= ld
> > drop *instantly* from 30,000 ops/s to flat zero, which would last=
> > for about 25 seconds before resuming exactly as it was before. > >
> > I wrote a little awk script to watch for this happening and run > > `procstat -k` on the nfsd process, and I saw that all but two of = the
> > service threads were idle.  The three nfsd threads that had = non-idle
> > kstacks were:
> >
> >   PID    TID COMM    &nbs= p;           TDNAME =             &nb= sp;KSTACK
> >   997 108481 nfsd       &= nbsp;        nfsd: master  &nb= sp;     mi_switch sleepq_timedwait _sleep nfsv4_lo= ck nfsrvd_dorpc nfssvc_program svc_run_internal svc_run nfsrvd_nfsd nfssvc_= nfsd sys_nfssvc amd64_syscall fast_syscall_common
> >   997 960918 nfsd       &= nbsp;        nfsd: service  &n= bsp;    mi_switch sleepq_timedwait _sleep nfsv4_lock nf= srv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfssvc_program svc_run_interna= l svc_thread_start fork_exit fork_trampoline
> >   997 962232 nfsd       &= nbsp;        nfsd: service  &n= bsp;    mi_switch _cv_wait txg_wait_synced_impl txg_wai= t_synced dmu_offset_next zfs_holey zfs_freebsd_ioctl vn_generic_copy_file_r= ange vop_stdcopy_file_range VOP_COPY_FILE_RANGE vn_copy_file_range nfsrvd_c= opy_file_range nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_star= t fork_exit fork_trampoline
> >
> > I'm suspicious of two things: first, the copy_file_range RPC; sec= ond,
> > the "master" nfsd thread is actually servicing an RPC which requi= res
> > obtaining a lock.  The "master" getting stuck while performi= ng client
> > RPCs is, I believe, the reason NFS service grinds to a halt when = a
> > client tries to write into a near-full filesystem, so this proble= m
> > would be more evidence that the dispatching function should not b= e
> > mixed with actual operations.  I don't know what the clients= are
> > doing, but is it possible that nfsrvd_copy_file_range is holding = a
> > lock that is needed by one or both of the other two threads?
> >
> > Near-term I could change nfsrvd_copy_file_range to just
> > unconditionally return NFSERR_NOTSUP and force the clients to fal= l
> > back, but I figured I would ask if anyone else has seen this.
> I have attached a little patch that should limit the server's Copy siz= e
> to vfs.nfsd.maxcopyrange (default of 10Mbytes).
> Hopefully this makes sure that the Copy does not take too long.
>
> You could try this instead of disabling Copy. It would be nice to know= if
> this is suffciient? (If not, I'll probably add a sysctl to disable Cop= y.)
I did a quick test without/with this patch,where I copied a 1Gbyte file.
Without this patch, the Copy RPCs mostly replied in just under 1sec
(which is what the flag requests), but took over 4sec for one of the Copy operations. This implies that one Read/Write of 1Mbyte on the server
took over 3 seconds.
I noticed the first Copy did over 600Mbytes, but the rest did about 100Mbyt= es
each and it was one of these 100Mbyte Copy operations that took over 4sec.<= br>
With the patch, there were a lot more Copy RPCs (as expected) of 10Mbytes each and they took a consistent 0.25-0.3sec to reply. (This is a test of a = local
mount on an old laptop, so nowhere near a server hardware config.)

So, the patch might be sufficient?

It would be nice to avoid disabling Copy, since it avoids reading the data<= br> into the client and then writing it back to the server.

I will probably commit both patches (10Mbyte clip of Copy size and
disabling Copy) to main soon, since I cannot say if clipping the size
of the Copy will always be sufficient.

Pleas let us know how trying these patches goes, rick

>
> rick
>
> >
> > -GAWollman
> >
> >




------=_Part_1591_1207250147.1709280020983-- From nobody Fri Mar 1 14:23:56 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TmVhc0Jtmz5CThp for ; Fri, 1 Mar 2024 14:24:16 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TmVhb1pwnz4653; Fri, 1 Mar 2024 14:24:15 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-6de3141f041so1570675b3a.0; Fri, 01 Mar 2024 06:24:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709303053; x=1709907853; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=DJ+K9tLDyDpLuMYSlLSB6p2LlqhG/4qzZu8ig/PeHbw=; b=GEVOn2R+W8pLgYlHudDo3YH/UmIopazPR5//TKgDspHgGak+cA9EvDU5tJ5ioRA5t0 uCjPxLWQYuS0Avt0tYIDOxWVXMKJ8fEbXGOgrKew1TTtEq+nr1A/YCvkp/FHsxMPUFpq ls4Mh+7IycYBBzZR1jTN46H68OK4TOdYFzSJvV4aq5j5oIiVre7S87HPf7zmPrHtCNHZ KSkQBjSUfRzYVk5WG4b0octRJk4ztgJ2d5YUSJV/tNpath+uDX0vLpQqijgXmPRfNtb1 inpE0Hx1RjoPplOGWz4Mwsp8DljgXKrLdRzbzJTFHl3aQg2lsS3vKzy1AhLAsKT4Smz1 +3ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709303053; x=1709907853; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DJ+K9tLDyDpLuMYSlLSB6p2LlqhG/4qzZu8ig/PeHbw=; b=NaX3GWxjmH/vToMmLOspu/WexG8qX99MBGnsz59WJqemAX5xVu6Oobi4NQk9oJV51F tz46SH2d6KeFIv1SifBY7fPD5WnspyAtv1U88Gyd9kClaxb3Uz7uxQwgJyBm70rui3tY ZKXhAEyxZqddULZY5ELXcNF/rgLb+qnyioGXJmQqLPUEH8PpGYshONAOtiuKskmzk4Tt SFh7AcYTC9PGMcDhasVoMwzAoVVlUf7or4qTmtQzcW7507rO3fuh08EFiUtcjImNpWQE UgzyCYiK6IEVS2uKwb5nAjSHtExIJCDTTSls6vOi+GRwGjn1A8OSe3hMU0G+X9D3ATX/ KiJQ== X-Forwarded-Encrypted: i=1; AJvYcCVLUW0WnnUCTZ+6yJ3Mk9JNtl5G2DVRe7oHqpsnH0zu/5Cjk9WURz3RSdF/OTGN1EJ1S5xSwR6K+OGQiFjDI3XJqKT3mCuIq39RKwVHKfS83hy/1aLBLvn/ X-Gm-Message-State: AOJu0YyDZLz8QeNCdP1F5S9SSCUAULue5kCdeSdY+2+nAS/TjEhedLe0 U7XANhIOTraLxVxkW3BtR7VED78xLgIuJTR9IpvsretCFgijPZ89bszHhRK8ucT5ov+e3JG2hh+ kax5vbUzJrcgx/AVDn6spRtNRQsc+rUo= X-Google-Smtp-Source: AGHT+IHoinZp6TsQpmLmZPsOOwuIuVunvM78NTIZonI2YPnPfczE5v9GY6U2g1QuVWzuns5/Eb4+EAd9rSQ1hUeDV2E= X-Received: by 2002:a05:6a21:328a:b0:1a0:e944:15b7 with SMTP id yt10-20020a056a21328a00b001a0e94415b7mr2038806pzb.5.1709303053035; Fri, 01 Mar 2024 06:24:13 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <1020651467.1592.1709280020993@localhost> In-Reply-To: <1020651467.1592.1709280020993@localhost> From: Rick Macklem Date: Fri, 1 Mar 2024 06:23:56 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Ronald Klop Cc: Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4TmVhb1pwnz4653 On Fri, Mar 1, 2024 at 12:00=E2=80=AFAM Ronald Klop = wrote: > > Interesting read. > > Would it be possible to separate locking for admin actions like a client= mounting an fs from traffic flowing for file operations? Well, the NFS server does not really have any concept of a mount. What I am referring to is the ClientID maintained for NFSv4 mounts, which all the open/lock/session/layout state hangs off of. For most cases, this state information can safely be accessed/modified via a mutex, but there are three exceptions: - creating a new ClientID (which is done by the ExchangeID operation) and typically happens when a NFS client does a mount. - delegation Recall (which only happens when delegations are enabled) One of the reasons delegations are not enabled by default on the FreeBSD server. - the DestroyClientID which is typically done by a NFS client during dismou= nt. For these cases, it is just too difficult to do them without sleeping. As such, there is a sleep lock which the nfsd threads normally acquire shar= ed when doing NFSv4 operations, but for the above cases the lock is aquired exclusive. - I had to give the exclusive lock priority over shared lock acquisition (it is a custom locking mechanism with assorted weirdnesses) because without that someone reported that new mounts took up to 1/2hr to occur. (The exclusive locker waited for 30min before all the other nfsd threads were not busy.) Because of this priority, once a nfsd thread requests the exclusive lock, all other nfsd threads executing NFSv4 RPCs block after releasing their shared lock, until the exclusive locker releases the exclusive lock. In summary, NFSv4 has certain advantages over NFSv3, but it comes with a lot of state complexity. It just is not feasible to manipulate all t= hat state with only mutex locking. rick > > Like ongoing file operations could have a read only view/copy of the moun= t table. Only new operations will have to wait. > But the mount never needs to wait for ongoing operations before locking t= he structure. > > Just a thought in the morning > > Regards, > Ronald. > > Van: Rick Macklem > Datum: 1 maart 2024 00:31 > Aan: Garrett Wollman > CC: stable@freebsd.org, rmacklem@freebsd.org > Onderwerp: Re: 13-stable NFS server hang > > On Wed, Feb 28, 2024 at 4:04PM Rick Macklem wrote: > > > > On Tue, Feb 27, 2024 at 9:30PM Garrett Wollman wrote: > > > > > > Hi, all, > > > > > > We've had some complaints of NFS hanging at unpredictable intervals. > > > Our NFS servers are running a 13-stable from last December, and > > > tonight I sat in front of the monitor watching `nfsstat -dW`. I was > > > able to clearly see that there were periods when NFS activity would > > > drop *instantly* from 30,000 ops/s to flat zero, which would last > > > for about 25 seconds before resuming exactly as it was before. > > > > > > I wrote a little awk script to watch for this happening and run > > > `procstat -k` on the nfsd process, and I saw that all but two of the > > > service threads were idle. The three nfsd threads that had non-idle > > > kstacks were: > > > > > > PID TID COMM TDNAME KSTACK > > > 997 108481 nfsd nfsd: master mi_switch sleepq= _timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal s= vc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > > > 997 960918 nfsd nfsd: service mi_switch sleepq= _timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc= nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > > 997 962232 nfsd nfsd: service mi_switch _cv_wa= it txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freeb= sd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RA= NGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program s= vc_run_internal svc_thread_start fork_exit fork_trampoline > > > > > > I'm suspicious of two things: first, the copy_file_range RPC; second, > > > the "master" nfsd thread is actually servicing an RPC which requires > > > obtaining a lock. The "master" getting stuck while performing client > > > RPCs is, I believe, the reason NFS service grinds to a halt when a > > > client tries to write into a near-full filesystem, so this problem > > > would be more evidence that the dispatching function should not be > > > mixed with actual operations. I don't know what the clients are > > > doing, but is it possible that nfsrvd_copy_file_range is holding a > > > lock that is needed by one or both of the other two threads? > > > > > > Near-term I could change nfsrvd_copy_file_range to just > > > unconditionally return NFSERR_NOTSUP and force the clients to fall > > > back, but I figured I would ask if anyone else has seen this. > > I have attached a little patch that should limit the server's Copy size > > to vfs.nfsd.maxcopyrange (default of 10Mbytes). > > Hopefully this makes sure that the Copy does not take too long. > > > > You could try this instead of disabling Copy. It would be nice to know = if > > this is suffciient? (If not, I'll probably add a sysctl to disable Copy= .) > I did a quick test without/with this patch,where I copied a 1Gbyte file. > > Without this patch, the Copy RPCs mostly replied in just under 1sec > (which is what the flag requests), but took over 4sec for one of the Copy > operations. This implies that one Read/Write of 1Mbyte on the server > took over 3 seconds. > I noticed the first Copy did over 600Mbytes, but the rest did about 100Mb= ytes > each and it was one of these 100Mbyte Copy operations that took over 4sec= . > > With the patch, there were a lot more Copy RPCs (as expected) of 10Mbytes > each and they took a consistent 0.25-0.3sec to reply. (This is a test of = a local > mount on an old laptop, so nowhere near a server hardware config.) > > So, the patch might be sufficient? > > It would be nice to avoid disabling Copy, since it avoids reading the dat= a > into the client and then writing it back to the server. > > I will probably commit both patches (10Mbyte clip of Copy size and > disabling Copy) to main soon, since I cannot say if clipping the size > of the Copy will always be sufficient. > > Pleas let us know how trying these patches goes, rick > > > > > rick > > > > > > > > -GAWollman > > > > > > > > ________________________________ > > > > From nobody Fri Mar 1 19:19:57 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TmdGD4HVYz5CvK7 for ; Fri, 1 Mar 2024 19:20:20 +0000 (UTC) (envelope-from juraj@lutter.sk) Received: from ns2.wilbury.net (ns2.wilbury.net [IPv6:2a01:b200:0:1:f816:3eff:fecd:13e6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "svc.wilbury.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TmdGC4VMPz4hlL for ; Fri, 1 Mar 2024 19:20:19 +0000 (UTC) (envelope-from juraj@lutter.sk) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=lutter.sk; spf=pass (mx1.freebsd.org: domain of juraj@lutter.sk designates 2a01:b200:0:1:f816:3eff:fecd:13e6 as permitted sender) smtp.mailfrom=juraj@lutter.sk Received: from smtpclient.apple (gw-upc.owhome.net [188.167.168.254]) (Authenticated sender: juraj@lutter.sk) by svc.wilbury.net (Postfix) with ESMTPSA id A394A6219C for ; Fri, 1 Mar 2024 20:20:07 +0100 (CET) From: Juraj Lutter Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: CPU hog in vnlru and arc_prune on 13-STABLE Message-Id: Date: Fri, 1 Mar 2024 20:19:57 +0100 To: FreeBSD stable X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.70 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[lutter.sk,none]; R_SPF_ALLOW(-0.20)[+ip6:2a01:b200:0:1:f816:3eff:fecd:13e6]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:44185, ipnet:2a01:b200::/32, country:SK]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; APPLE_MAILER_COMMON(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4TmdGC4VMPz4hlL Hi, I have a fresh (git b9880e247b26) 13.3-STABLE running in bhyve with 2 = vCPU and 10G of RAM. I=E2=80=99ve put some load on it (building lang/rust from ports outside = poudriere, using make -j2) and the system started to respond very slowly. I did break the make process and I was able to look into top where there = was: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU = COMMAND 0 root -8 - 0B 1808K CPU1 1 35:53 99.79% = kernel{arc_prune} 15 root 20 - 0B 16K RUN 0 34:19 99.35% vnlru The high CPU usage remains even after 30 minutes after the make was = stopped. How can I narrow down the root cause? For the record, the kernel is = GENERIC. Thanks otis =E2=80=94 Juraj Lutter XMPP: juraj (at) lutter.sk GSM: +421907986576 From nobody Fri Mar 1 19:38:50 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tmdgq0nxdz5CwjL for ; Fri, 1 Mar 2024 19:39:03 +0000 (UTC) (envelope-from juraj@lutter.sk) Received: from ns2.wilbury.net (ns2.wilbury.net [IPv6:2a01:b200:0:1:f816:3eff:fecd:13e6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "svc.wilbury.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tmdgp2tjTz4lBf for ; Fri, 1 Mar 2024 19:39:02 +0000 (UTC) (envelope-from juraj@lutter.sk) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=lutter.sk; spf=pass (mx1.freebsd.org: domain of juraj@lutter.sk designates 2a01:b200:0:1:f816:3eff:fecd:13e6 as permitted sender) smtp.mailfrom=juraj@lutter.sk Received: from smtpclient.apple (gw-upc.owhome.net [188.167.168.254]) (Authenticated sender: juraj@lutter.sk) by svc.wilbury.net (Postfix) with ESMTPSA id 95B1261F91 for ; Fri, 1 Mar 2024 20:39:00 +0100 (CET) From: Juraj Lutter Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: CPU hog in vnlru and arc_prune on 13-STABLE Date: Fri, 1 Mar 2024 20:38:50 +0100 References: To: FreeBSD stable In-Reply-To: Message-Id: <71A074C6-6C33-45BA-A42C-B176DEF909EF@lutter.sk> X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.69 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.992]; DMARC_POLICY_ALLOW(-0.50)[lutter.sk,none]; R_SPF_ALLOW(-0.20)[+ip6:2a01:b200:0:1:f816:3eff:fecd:13e6:c]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; ASN(0.00)[asn:44185, ipnet:2a01:b200::/32, country:SK]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ONE(0.00)[1]; RCVD_VIA_SMTP_AUTH(0.00)[]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4Tmdgp2tjTz4lBf > On 1 Mar 2024, at 20:19, Juraj Lutter wrote: >=20 > Hi, >=20 > I have a fresh (git b9880e247b26) 13.3-STABLE running in bhyve with 2 = vCPU and 10G of RAM. >=20 > I=E2=80=99ve put some load on it (building lang/rust from ports = outside poudriere, using make -j2) and the system > started to respond very slowly. >=20 > I did break the make process and I was able to look into top where = there was: >=20 > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU = COMMAND > 0 root -8 - 0B 1808K CPU1 1 35:53 99.79% = kernel{arc_prune} > 15 root 20 - 0B 16K RUN 0 34:19 99.35% = vnlru >=20 > The high CPU usage remains even after 30 minutes after the make was = stopped. >=20 > How can I narrow down the root cause? For the record, the kernel is = GENERIC. After manually issuing =E2=80=9Csync=E2=80=9D, the CPU hog stopped and = the machine is responding normally now. It still remains unclear to me why that situation could happen, though. =E2=80=94 Juraj Lutter XMPP: juraj (at) lutter.sk GSM: +421907986576 From nobody Fri Mar 1 19:43:29 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TmdnG454Lz5Cx0m for ; Fri, 1 Mar 2024 19:43:46 +0000 (UTC) (envelope-from lexi@le-fay.org) Received: from thyme.eden.le-Fay.ORG (THYME.EDEN.LE-FAY.ORG [81.187.47.194]) by mx1.freebsd.org (Postfix) with ESMTP id 4TmdnF3DZcz4mWK for ; Fri, 1 Mar 2024 19:43:45 +0000 (UTC) (envelope-from lexi@le-fay.org) Authentication-Results: mx1.freebsd.org; none Received: from iris.eden.le-Fay.ORG (IRIS.EDEN.LE-FAY.ORG [IPv6:2001:8b0:aab5:106:3::6]) by thyme.eden.le-Fay.ORG (Postfix) with ESMTP id F2101AB; Fri, 1 Mar 2024 19:43:29 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=le-fay.org; s=thyme; t=1709322210; bh=cyZD5sJxn/EehpfDuBYBHskdFxm/9YmFBLJQRoXBRZ4=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=ka7aKoW2ynk8a2fwoTFMGzOdxMfSclGSEMEIinHfCLSgVya0N4vA8pxY7Ki//d+x0 hcbJbY1/hRnCfCUAvyiLT1R0wfb25frkl27t7eSVxXUTrxCVtTsaOfTIfaYR+f5ir9 utcMujCkuXqSqqjzwNZ3JB8jFSgTSiEEf2XvSLQ8= Received: from ilythia.eden.le-fay.org (ILYTHIA.EDEN.LE-FAY.ORG [IPv6:2001:8b0:aab5:106:3::10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by iris.eden.le-Fay.ORG (Postfix) with ESMTPSA id CEE712C041C; Fri, 1 Mar 2024 19:43:29 +0000 (GMT) Date: Fri, 1 Mar 2024 19:43:29 +0000 From: Lexi Winter To: Juraj Lutter Cc: FreeBSD stable Subject: Re: CPU hog in vnlru and arc_prune on 13-STABLE Message-ID: Mail-Followup-To: Juraj Lutter , FreeBSD stable References: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="9hYQUd4wgAE5mdej" Content-Disposition: inline In-Reply-To: X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:20712, ipnet:81.187.0.0/16, country:GB] X-Rspamd-Queue-Id: 4TmdnF3DZcz4mWK --9hYQUd4wgAE5mdej Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Juraj Lutter: > I have a fresh (git b9880e247b26) 13.3-STABLE running in bhyve with 2 vCP= U and 10G of RAM. >=20 > I=E2=80=99ve put some load on it (building lang/rust from ports outside p= oudriere, using make -j2) and the system > started to respond very slowly. >=20 > I did break the make process and I was able to look into top where there = was: >=20 > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 0 root -8 - 0B 1808K CPU1 1 35:53 99.79% kernel= {arc_prune} > 15 root 20 - 0B 16K RUN 0 34:19 99.35% vnlru this could be [0]: "High CPU usage by arc_prune; analysis and fix". [0] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --9hYQUd4wgAE5mdej Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCAAdFiEEuwt6MaPcv/+Mo+ftDHqbqZ41x5kFAmXiL94ACgkQDHqbqZ41 x5mZoQv/ctI8kZYBwX37vKgcQQzp23i1z2VsOIasiY0ubiiFpRWhwPOWCiL0mSWp 29nB5D+d1zxgv2n0fAvnsx1GXfktnlYy9eMECC42Yw2Bop61JsBHCuH83kS08Kvk i25q5VU4keRf+YstHtqncaqzcpsX0VrQGhh8zyUTK2q/3otxp1/ETJ1WLF8oazZO +HDVacB5mTj4sDOhrnChn8ZFTiut27jAfMpr0E7KF6Xkw2hbRymNHU8wDerEzrwo OkI0qTG3KjrvlH+cHbDx17JhBPkxqAHuEe694U/gTG4H13kIYmlt9/ZFTuff6y+O rjun633AFN4N4yA9fsGJoVRPiGUyBcvdF2y+Kr+ZWqwObKhmJfYlWC7PwROcP72o 9ajCo4fZ0xBK15qjv04n15NwKJnYfCihrey0PNLx5roEzVxVnseB1QnZ2sg3gQJW wrEbTeIK/pNDxpfaam7M9gkngwqHDA0O8W6oU4b4EbPZLD1OZl5XY5GlzlrUUY2W GEHzMvlQ =Xe2w -----END PGP SIGNATURE----- --9hYQUd4wgAE5mdej-- From nobody Fri Mar 1 19:47:47 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tmdt70XNDz5Cwsh for ; Fri, 1 Mar 2024 19:47:59 +0000 (UTC) (envelope-from juraj@lutter.sk) Received: from ns2.wilbury.net (ns2.wilbury.net [IPv6:2a01:b200:0:1:f816:3eff:fecd:13e6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "svc.wilbury.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tmdt66gz1z4nNP for ; Fri, 1 Mar 2024 19:47:58 +0000 (UTC) (envelope-from juraj@lutter.sk) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (gw-upc.owhome.net [188.167.168.254]) (Authenticated sender: juraj@lutter.sk) by svc.wilbury.net (Postfix) with ESMTPSA id ADA9961F91; Fri, 1 Mar 2024 20:47:57 +0100 (CET) Content-Type: text/plain; charset=utf-8 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: CPU hog in vnlru and arc_prune on 13-STABLE From: Juraj Lutter In-Reply-To: Date: Fri, 1 Mar 2024 20:47:47 +0100 Cc: FreeBSD stable Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Lexi Winter X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:44185, ipnet:2a01:b200::/32, country:SK] X-Rspamd-Queue-Id: 4Tmdt66gz1z4nNP > On 1 Mar 2024, at 20:43, Lexi Winter wrote: >=20 > Juraj Lutter: >> I have a fresh (git b9880e247b26) 13.3-STABLE running in bhyve with 2 = vCPU and 10G of RAM. >>=20 >> I=E2=80=99ve put some load on it (building lang/rust from ports = outside poudriere, using make -j2) and the system >> started to respond very slowly. >>=20 >> I did break the make process and I was able to look into top where = there was: >>=20 >> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU = COMMAND >> 0 root -8 - 0B 1808K CPU1 1 35:53 99.79% = kernel{arc_prune} >> 15 root 20 - 0B 16K RUN 0 34:19 99.35% = vnlru >=20 > this could be [0]: "High CPU usage by arc_prune; analysis and fix". >=20 > [0] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 Indeed, the symptoms are very similar. Thanks for the pointer. =E2=80=94 Juraj Lutter XMPP: juraj (at) lutter.sk GSM: +421907986576 From nobody Sat Mar 2 06:51:22 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tmwc50Tzdz5C9HF for ; Sat, 2 Mar 2024 06:51:49 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tmwc41K0pz42cW; Sat, 2 Mar 2024 06:51:48 +0000 (UTC) (envelope-from kib@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kib@freebsd.org) smtp.mailfrom=kib@freebsd.org Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 4226pM3r013465; Sat, 2 Mar 2024 08:51:25 +0200 (EET) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 4226pM3r013465 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 4226pMh7013464; Sat, 2 Mar 2024 08:51:22 +0200 (EET) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Sat, 2 Mar 2024 08:51:22 +0200 From: Konstantin Belousov To: Rick Macklem Cc: Ronald Klop , Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Subject: Re: 13-stable NFS server hang Message-ID: References: <1020651467.1592.1709280020993@localhost> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: - X-Spamd-Result: default: False [-1.50 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; TAGGED_RCPT(0.00)[]; ARC_NA(0.00)[]; HAS_XAW(0.00)[]; FREEFALL_USER(0.00)[kib]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; MLMMJ_DEST(0.00)[stable@freebsd.org]; MISSING_XM_UA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCPT_COUNT_FIVE(0.00)[5] X-Rspamd-Queue-Id: 4Tmwc41K0pz42cW On Fri, Mar 01, 2024 at 06:23:56AM -0800, Rick Macklem wrote: > On Fri, Mar 1, 2024 at 12:00 AM Ronald Klop wrote: > > > > Interesting read. > > > > Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations? > Well, the NFS server does not really have any concept of a mount. > What I am referring to is the ClientID maintained for NFSv4 mounts, > which all the open/lock/session/layout state hangs off of. > > For most cases, this state information can safely be accessed/modified > via a mutex, but there are three exceptions: > - creating a new ClientID (which is done by the ExchangeID operation) > and typically happens when a NFS client does a mount. > - delegation Recall (which only happens when delegations are enabled) > One of the reasons delegations are not enabled by default on the > FreeBSD server. > - the DestroyClientID which is typically done by a NFS client during dismount. > For these cases, it is just too difficult to do them without sleeping. > As such, there is a sleep lock which the nfsd threads normally acquire shared > when doing NFSv4 operations, but for the above cases the lock is aquired > exclusive. > - I had to give the exclusive lock priority over shared lock > acquisition (it is a > custom locking mechanism with assorted weirdnesses) because without > that someone reported that new mounts took up to 1/2hr to occur. > (The exclusive locker waited for 30min before all the other nfsd threads > were not busy.) > Because of this priority, once a nfsd thread requests the exclusive lock, > all other nfsd threads executing NFSv4 RPCs block after releasing their > shared lock, until the exclusive locker releases the exclusive lock. Normal lockmgr locks + TDP_DEADLKTREAT private thread flag provide the property of pref. exclusive waiters in presence of the shared waiters. I think this is what you described above. From nobody Sat Mar 2 13:40:08 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tn5gc540Wz5ClPy for ; Sat, 2 Mar 2024 13:40:28 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tn5gc3NDjz4jyZ; Sat, 2 Mar 2024 13:40:28 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1dc09556599so28515295ad.1; Sat, 02 Mar 2024 05:40:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709386826; x=1709991626; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=1slmapmEdIkS16Pqi+j8NKrv8fgkpwTTKhqXYr7ItA8=; b=m4FA+7kmPARQ7mrLbmeDyEt4f3q+rQBc33/dh8E2r9Ciz1LfH89e5+H0f2JqQuz5iC BZY0y0JD2sJHzu/NRCvfU26pWcWQKKi5zaq0qtAJur+vPApa+fL5OstaheVlK9TNEk5w 3TWdrWKn5iEg7dASVIPoZvjME6fRdTuG6lYvh1ykTSzdx5ohP94PBNQbQD/0c2jKTDfk klnKQBUfEiEqZr8f0GGX3IenEyNn84AdTelAWqdaZFWUXZ2g7rv6bhBQE4iOnswXyQH5 c1nClfaRE73ljDGbauJXZ9AYF35jTZi9whcnby/D0syEkpLLfS/ugrIHfuqOd5Cx+ZNi pNlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709386826; x=1709991626; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1slmapmEdIkS16Pqi+j8NKrv8fgkpwTTKhqXYr7ItA8=; b=hDJptOfZOJtRjkEwtRhVUIQQBvQdf4oysZg0FQdTdsh9UXixx3i7sxi9zFFeMuyKN1 0g4cBi4pB9sCXhkrlbTFzXOU9XJHgrGgadRcZuAFulshmX7UQHetTur3ADVGFEXo8H6r yA75QKh+hEdCnOXillg3fQzKW4xpfuY6szaqVbBahv9WsE+MUyg6Y6isnpu+Nbx9fnwe kC8kvUOm4niN/b8QUaoVUnfiUh9JH5Zs8VSlv3Trfw3/JILMFZYKmgI3C6GTinyrZEhr EgrxCfvk5lrrCWWTND+yNBlkQrOWJCIr59/h4sstb2jiDWqriDcBxoe5U+G3I8bJ6mOc fOfA== X-Forwarded-Encrypted: i=1; AJvYcCWWYs2+IoMAyV+4MiUg7ro6UBpB/0qCpXf5LclZ4DCaEqKDiMap0eEfRoBO/0vz9CbNCVapd3EhDqQcQlkC3tKZRNTAIV9sWSJoml4FQ6eE9vy2ndhisZiY X-Gm-Message-State: AOJu0Yw0suNYKAWdrXpgtWkJaFlFdqZDRdJCnzxrtRWmv5qFeMD7cewx 2E9HoppjpVaeXfoh6vQSJ/eHYOgmwIcSBOoiVaJ/jxqkFiwbYhQrvhFa7dX8TdaYFZCs/cDLy4Y rBOwkd2x7XqdsDuYloV1JNPJ2TupAU+4= X-Google-Smtp-Source: AGHT+IFoB+jF2wwvckyMraZ9NPPw+nMbi3VhLgZTi6VCI2bbbThyUIkBybKrR9OQnvA1OMikLdYfwu+0x7soU1G+1O4= X-Received: by 2002:a17:902:82c7:b0:1dc:affb:1f50 with SMTP id u7-20020a17090282c700b001dcaffb1f50mr4399281plz.47.1709386826006; Sat, 02 Mar 2024 05:40:26 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <1020651467.1592.1709280020993@localhost> In-Reply-To: From: Rick Macklem Date: Sat, 2 Mar 2024 05:40:08 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Konstantin Belousov Cc: Ronald Klop , Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Tn5gc3NDjz4jyZ On Fri, Mar 1, 2024 at 10:51=E2=80=AFPM Konstantin Belousov wrote: > > CAUTION: This email originated from outside of the University of Guelph. = Do not click links or open attachments unless you recognize the sender and = know the content is safe. If in doubt, forward suspicious emails to IThelp@= uoguelph.ca. > > > On Fri, Mar 01, 2024 at 06:23:56AM -0800, Rick Macklem wrote: > > On Fri, Mar 1, 2024 at 12:00=E2=80=AFAM Ronald Klop wrote: > > > > > > Interesting read. > > > > > > Would it be possible to separate locking for admin actions like a cl= ient mounting an fs from traffic flowing for file operations? > > Well, the NFS server does not really have any concept of a mount. > > What I am referring to is the ClientID maintained for NFSv4 mounts, > > which all the open/lock/session/layout state hangs off of. > > > > For most cases, this state information can safely be accessed/modified > > via a mutex, but there are three exceptions: > > - creating a new ClientID (which is done by the ExchangeID operation) > > and typically happens when a NFS client does a mount. > > - delegation Recall (which only happens when delegations are enabled) > > One of the reasons delegations are not enabled by default on the > > FreeBSD server. > > - the DestroyClientID which is typically done by a NFS client during di= smount. > > For these cases, it is just too difficult to do them without sleeping. > > As such, there is a sleep lock which the nfsd threads normally acquire = shared > > when doing NFSv4 operations, but for the above cases the lock is aquire= d > > exclusive. > > - I had to give the exclusive lock priority over shared lock > > acquisition (it is a > > custom locking mechanism with assorted weirdnesses) because without > > that someone reported that new mounts took up to 1/2hr to occur. > > (The exclusive locker waited for 30min before all the other nfsd thre= ads > > were not busy.) > > Because of this priority, once a nfsd thread requests the exclusive l= ock, > > all other nfsd threads executing NFSv4 RPCs block after releasing the= ir > > shared lock, until the exclusive locker releases the exclusive lock. > Normal lockmgr locks + TDP_DEADLKTREAT private thread flag provide the > property of pref. exclusive waiters in presence of the shared waiters. > I think this is what you described above. It also has some weird properties, like if there are multiple requestors for the exclusive lock, once one thread gets it (the threads are nfsd worke= r threads and indistinct), the others that requested an exclusive thread are unblocked without the lock being issued to them. They then check if the exclusive lock is still needed (usually not, since the other thread has dealt with the case where it was needed) and then they can acquire a shared lock. Without this, there were cases where several threads would acquire the exclusive lock and then discover that the lock was not needed and just release it again. It also uses an assortment of weird flags/call args. rick From nobody Sat Mar 2 14:12:40 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tn6P345cvz5CJsf for ; Sat, 2 Mar 2024 14:12:55 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tn6P24N0dz4pvY; Sat, 2 Mar 2024 14:12:54 +0000 (UTC) (envelope-from kib@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kib@freebsd.org) smtp.mailfrom=kib@freebsd.org Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 422ECew7020064; Sat, 2 Mar 2024 16:12:43 +0200 (EET) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 422ECew7020064 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 422ECeNt020063; Sat, 2 Mar 2024 16:12:40 +0200 (EET) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Sat, 2 Mar 2024 16:12:40 +0200 From: Konstantin Belousov To: Rick Macklem Cc: Ronald Klop , Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Subject: Re: 13-stable NFS server hang Message-ID: References: <1020651467.1592.1709280020993@localhost> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: - X-Spamd-Result: default: False [-1.50 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; TAGGED_RCPT(0.00)[]; ARC_NA(0.00)[]; HAS_XAW(0.00)[]; FREEFALL_USER(0.00)[kib]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; MLMMJ_DEST(0.00)[stable@freebsd.org]; MISSING_XM_UA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCPT_COUNT_FIVE(0.00)[5] X-Rspamd-Queue-Id: 4Tn6P24N0dz4pvY On Sat, Mar 02, 2024 at 05:40:08AM -0800, Rick Macklem wrote: > On Fri, Mar 1, 2024 at 10:51 PM Konstantin Belousov wrote: > > > > CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca. > > > > > > On Fri, Mar 01, 2024 at 06:23:56AM -0800, Rick Macklem wrote: > > > On Fri, Mar 1, 2024 at 12:00 AM Ronald Klop wrote: > > > > > > > > Interesting read. > > > > > > > > Would it be possible to separate locking for admin actions like a client mounting an fs from traffic flowing for file operations? > > > Well, the NFS server does not really have any concept of a mount. > > > What I am referring to is the ClientID maintained for NFSv4 mounts, > > > which all the open/lock/session/layout state hangs off of. > > > > > > For most cases, this state information can safely be accessed/modified > > > via a mutex, but there are three exceptions: > > > - creating a new ClientID (which is done by the ExchangeID operation) > > > and typically happens when a NFS client does a mount. > > > - delegation Recall (which only happens when delegations are enabled) > > > One of the reasons delegations are not enabled by default on the > > > FreeBSD server. > > > - the DestroyClientID which is typically done by a NFS client during dismount. > > > For these cases, it is just too difficult to do them without sleeping. > > > As such, there is a sleep lock which the nfsd threads normally acquire shared > > > when doing NFSv4 operations, but for the above cases the lock is aquired > > > exclusive. > > > - I had to give the exclusive lock priority over shared lock > > > acquisition (it is a > > > custom locking mechanism with assorted weirdnesses) because without > > > that someone reported that new mounts took up to 1/2hr to occur. > > > (The exclusive locker waited for 30min before all the other nfsd threads > > > were not busy.) > > > Because of this priority, once a nfsd thread requests the exclusive lock, > > > all other nfsd threads executing NFSv4 RPCs block after releasing their > > > shared lock, until the exclusive locker releases the exclusive lock. > > Normal lockmgr locks + TDP_DEADLKTREAT private thread flag provide the > > property of pref. exclusive waiters in presence of the shared waiters. > > I think this is what you described above. > It also has some weird properties, like if there are multiple requestors > for the exclusive lock, once one thread gets it (the threads are nfsd worker > threads and indistinct), the others that requested an exclusive thread are > unblocked without the lock being issued to them. This sounds to me as LK_SLEEPFAIL feature of lockmgr. Do not underestimate the amount of weird features in it. > They then check if the exclusive lock is still needed (usually not, since > the other thread has dealt with the case where it was needed) and > then they can acquire a shared lock. > Without this, there were cases where several threads would acquire > the exclusive lock and then discover that the lock was not needed and > just release it again. > > It also uses an assortment of weird flags/call args. > > rick > From nobody Sat Mar 2 14:52:58 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tn7Hf2qqdz5CNBc for ; Sat, 2 Mar 2024 14:53:18 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tn7Hd6nZmz4vdB; Sat, 2 Mar 2024 14:53:17 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1032.google.com with SMTP id 98e67ed59e1d1-29a2545a1e7so2285803a91.2; Sat, 02 Mar 2024 06:53:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709391196; x=1709995996; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=iwILxCpl00VkleF46SMQTU7NI1hQzhRkNzAq2ZC8OCc=; b=lxFmtXCHVZvxE+h9WO2Iw2zEUDMM+LXT/K4GYLEG2fyTPEpa6GVzmbzE1QXBUPU5+7 +6ZnkK9ql3oXlyoGZQuaRLrRk8dwYS2vpwVaSPtDqo5snAtcQhoVZY6324kaLzDVFBhW Axp1ROliUg0oLiwkDvs2epnadL4+U23lC9tt0Re1PLkBKOzkgRVJO8lfjtNEVwZnL0mX J+BzoiQ4zCCvRRr3QrQq07JsP+zc2aGRD8GQ1mY6UZ/uUCULIHM2jg2Vxy+sWq8FsyE1 qG4iwJLLXHGKEvMoJxI/bv+taRvrGLBi2JLUvu+J42uZVThyLLtaLyI/TlAeNyWizqdn Erfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709391196; x=1709995996; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iwILxCpl00VkleF46SMQTU7NI1hQzhRkNzAq2ZC8OCc=; b=SP6b7+xX/29XBYVjEE7IBkFipzNjzn3LAsbsh9McA9E6hwwbo6BNiKc+BM8gP91Fiy P7YRO9j6thVB6IzyW1tUnsZ9dXGExZp6TFoXnij0FvvOOhW7PMbLHk1YFKq2TpsVw9yL e+TkLZRxO37mqFhWEuKRly0ww4CCRGpogzWX5aVEQqTGubCjao3Ng/LD7U/X8l2NnfWP QKOL0vJgV2JYfBQQOLvbJIwkb9whGhMDX7i4KdEd/EbIAbm6XD3WPPWcpejSMqADyxW+ 7fFCijc3Mv62kUVi2xRcgy5iiVN+0+7O5hfnBAMJ4oaaRG1hrIaUs4dnh0zDCoaajkrB EIZA== X-Forwarded-Encrypted: i=1; AJvYcCVodbznGNy69D7FIHUnBmTeXeQEY2g5w9mY4YaYarbGt00sbkG+nXQHNv3QBtg9aw/nIqsJYGzXUV14DvWsg+MU4UxZRuWGLo8qV5TxgZ3J/FIOMAJGM7yC X-Gm-Message-State: AOJu0Yzr2jyRpOPWhTnG/c34OYMedwm+SotNvTWeV6EFdEYnFfPM17BU 5wZ4funzRXRakfiyzgRgbu0MddkDPjqCd3YenHXU2SOx8hjrRcU8q3taOTR1yF8ZeHT7n41Jt/Q 3vpJ8ZXPkK7+QTaPwpHdxomz5vVlRImk= X-Google-Smtp-Source: AGHT+IFT2eGovaUyeUCDHdgTmeLosPF3mQae1OBTtJxTsdedjK/Axv/ryRxH0xYSXjKniYE9aQdpd0WvpASPWs7GDyk= X-Received: by 2002:a17:90a:4211:b0:29a:a1cd:da65 with SMTP id o17-20020a17090a421100b0029aa1cdda65mr3806220pjg.43.1709391195794; Sat, 02 Mar 2024 06:53:15 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <1020651467.1592.1709280020993@localhost> In-Reply-To: From: Rick Macklem Date: Sat, 2 Mar 2024 06:52:58 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Konstantin Belousov Cc: Ronald Klop , Garrett Wollman , stable@freebsd.org, rmacklem@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Tn7Hd6nZmz4vdB On Sat, Mar 2, 2024 at 6:13=E2=80=AFAM Konstantin Belousov wrote: > > CAUTION: This email originated from outside of the University of Guelph. = Do not click links or open attachments unless you recognize the sender and = know the content is safe. If in doubt, forward suspicious emails to IThelp@= uoguelph.ca. > > > On Sat, Mar 02, 2024 at 05:40:08AM -0800, Rick Macklem wrote: > > On Fri, Mar 1, 2024 at 10:51=E2=80=AFPM Konstantin Belousov wrote: > > > > > > CAUTION: This email originated from outside of the University of Guel= ph. Do not click links or open attachments unless you recognize the sender = and know the content is safe. If in doubt, forward suspicious emails to ITh= elp@uoguelph.ca. > > > > > > > > > On Fri, Mar 01, 2024 at 06:23:56AM -0800, Rick Macklem wrote: > > > > On Fri, Mar 1, 2024 at 12:00=E2=80=AFAM Ronald Klop wrote: > > > > > > > > > > Interesting read. > > > > > > > > > > Would it be possible to separate locking for admin actions like = a client mounting an fs from traffic flowing for file operations? > > > > Well, the NFS server does not really have any concept of a mount. > > > > What I am referring to is the ClientID maintained for NFSv4 mounts, > > > > which all the open/lock/session/layout state hangs off of. > > > > > > > > For most cases, this state information can safely be accessed/modif= ied > > > > via a mutex, but there are three exceptions: > > > > - creating a new ClientID (which is done by the ExchangeID operatio= n) > > > > and typically happens when a NFS client does a mount. > > > > - delegation Recall (which only happens when delegations are enable= d) > > > > One of the reasons delegations are not enabled by default on the > > > > FreeBSD server. > > > > - the DestroyClientID which is typically done by a NFS client durin= g dismount. > > > > For these cases, it is just too difficult to do them without sleepi= ng. > > > > As such, there is a sleep lock which the nfsd threads normally acqu= ire shared > > > > when doing NFSv4 operations, but for the above cases the lock is aq= uired > > > > exclusive. > > > > - I had to give the exclusive lock priority over shared lock > > > > acquisition (it is a > > > > custom locking mechanism with assorted weirdnesses) because witho= ut > > > > that someone reported that new mounts took up to 1/2hr to occur. > > > > (The exclusive locker waited for 30min before all the other nfsd = threads > > > > were not busy.) > > > > Because of this priority, once a nfsd thread requests the exclusi= ve lock, > > > > all other nfsd threads executing NFSv4 RPCs block after releasing= their > > > > shared lock, until the exclusive locker releases the exclusive lo= ck. > > > Normal lockmgr locks + TDP_DEADLKTREAT private thread flag provide th= e > > > property of pref. exclusive waiters in presence of the shared waiters= . > > > I think this is what you described above. > > It also has some weird properties, like if there are multiple requestor= s > > for the exclusive lock, once one thread gets it (the threads are nfsd w= orker > > threads and indistinct), the others that requested an exclusive thread = are > > unblocked without the lock being issued to them. > This sounds to me as LK_SLEEPFAIL feature of lockmgr. > Do not underestimate the amount of weird features in it. Yep, sounds like it. I should take a look to see if lockmgr will work instead of the "rolled my own". I should also take another look at new client creation, to see if there is = a way to do it that doesn't require the exclusive lock (a lot of that code is 20years old now). rick > > > They then check if the exclusive lock is still needed (usually not, sin= ce > > the other thread has dealt with the case where it was needed) and > > then they can acquire a shared lock. > > Without this, there were cases where several threads would acquire > > the exclusive lock and then discover that the lock was not needed and > > just release it again. > > > > It also uses an assortment of weird flags/call args. > > > > rick > > From nobody Sat Mar 2 15:06:28 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tn7Zy42xcz5CPHG for ; Sat, 2 Mar 2024 15:06:34 +0000 (UTC) (envelope-from filippomore@yahoo.com) Received: from sonic301-31.consmr.mail.ne1.yahoo.com (sonic301-31.consmr.mail.ne1.yahoo.com [66.163.184.200]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tn7Zx5sJ8z3xnt for ; Sat, 2 Mar 2024 15:06:33 +0000 (UTC) (envelope-from filippomore@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=TqXYqWys; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of filippomore@yahoo.com designates 66.163.184.200 as permitted sender) smtp.mailfrom=filippomore@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709391989; bh=hklWZbufILKvamIB/dTziOztw0vx3SjOmd9nKxksvYA=; h=Date:From:To:Subject:References:From:Subject:Reply-To; b=TqXYqWysOQiQklDzCekI7y5wq+zrfHll3UlBZPsyHaWZjVIyUzZZ1RX01JD/MuHPqwZ5RRKdR/MgMd1EsMFEgG0yTsKo38J8ITv1OSKviCl3Dj81drC5pp5dur7f0oqKXMq7hfv9vQsHv6z5A76k0usmB+MVso5D5nuNT073GoRHgGeZrkBq9HBMiV7cTve7SoDcMoA0eTbsi06s7qczBg1WK34XYCEg4GsDOwL476MiC0ONE6oC+nO4n2es4k4L/zAW3m4Lu28IZ6w2pUATTW3giisj1SMGwgb9DrkZbaNZJCLrf+SW6Dp6+Kcl+2zoLHUDEi7z6MeIKmJyHhyQFw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709391989; bh=5Y8SNs9i5Z0GHvxEjprStyHtdd8Myc8k0rMDw6PxOK6=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=WoU25DBkQiK8hLfIJLSyXP8zJ8UqX9/y9FD9Adr8f9j1OZ+3uQj5uCFRWw8ZduQJIr67xrmqfT5zmZms5Mw5CGWQJQlcG1q524svCP9ymW9l3vTfz3DgvwM9tuV93/GjAG+gEcNERf8BvdhJ1lYSvFSXMZ7/KAyAMxUWrD3VPgSwb+oe+urk0mC7Ufi4+rgXkJdke7IhOnpDsDDcXAdHeycJ3b3/RHLib8Rm0WRVDOVw+/9ytHBvbhCm3nGNOsfrO7tGYseVQnqM9P3640Mr0Hv/WSXilj3yFPSnmBYv21wxPgLInSGSjQreXjigQr1ddFBQ3o/SZ7nXnRXqSKXByQ== X-YMail-OSG: F0U3xnkVM1kQb4i_vT.pKwHwybRBBDLu.GUpO2iIXsAGE8U9t3OVUbpUYpcyDHK hqHZpmnFE13eiOdrcijaSfld9TsUfJyPCa0lFr8t3fG0LIjmByWEm8C6IwMXHm7V1sLnC2VnySZh 8.Wf1XHBUoQSBLHtFIHbo8mb0t3uAzOGFGJfORqwXlsmCHC1L6rERe9J_fspZPGswXuiiK5fkpwk Z2ivJlu92wQyPBaU4Ei8ftJJz12lFdOLsiIjd36BjGkKTI_gZAQ2SQS9B3FEpzBnTh2J6TJzmDSI kMAd.bjbsOYF5R67MFoFm6fOrpHM4j693xLl9Z9o8omYpzJmDelfy.qrVZquvEFXBP_kLceOXCgm aQskmaPVAZ8rm_goEPMjV6Ww1dFjQDZA4pUPPZ47p6U0lMtWs2u_NKepTsFpOHgaXuogRzfqmqY6 OL6y73IbQLpFOtaKUjQ2CX3mM7cTd9rNRrNgLKGkoYjqrU7s8MrLrAOmjMYLpURUgAHZJmCVdUdT 33xDBgKeBDhV6iO2G.P0f6L1JARCM0kQo5kXoijlssqyS9cfCmtpjBNY.fZrgCNwkkNjHeklx_s0 47rWNPjV1iWH2lpaiAO5_Jxz7gVn9X4C2ds3gIBQOqR9Ta6xbHVO9V3oo.WxttcEMt_81Jhsqymd GghPixY8p21MWz06uRCC.KjPYj2nGE1StlFiryXbfUaxccIXzsynAqmkglN5ekQGjl6IXdfyoNGz bLLM0flpXxKax5w_56r46DRnMUK66JE_nlOIhOlpSjG7aQsitF7o3_JicgFFpEkHzmVXu8K2F4ke TGTMJNh3084S0bJd1YHv4qq.mjcxrItj54mDfNcGV95BkLHHNIFYXzs8wJJpzKf.paQvUIlYCUBW i.oHy_BLPwTgoYt8s8R6yr3BoGkoAsf3MQxHZGghwVJf5JQFJrTheVBIw4RUI5Ev9_6kShuVmoDc 1IxYWEFeoKhzxDliO1nVrtPglha_aNb6bcZ2Y7wa9TCMjcJbOIaAZDxPxM5e0PVpIr0MNZwpVWW4 dmAHknp8i.YufNgNuLA.QK1g1W3HnIsL5jSKTQsWjSmxGrY_Tu.6Cd0Bj8l9B6yx2qMEJ6MvmxuQ 8k4rJzQ1j3wlsJd7IcI614GkOy6_M4t0FgrfrkMqczGDHxmIfbrR6e8I8piZPhIxvLImtQkywSY8 f9Ig7JLPszfku4ac1cPSrOi_EA27qQ8FMc7njI2pcguT8M4u7XouddV3SryWX3w4ZIGtE8Mkxnew aYMPic5pqxsOq9u4sMoTRjeHtg0MnMmBhlWcKZpQQmGRyRiDn2cbl6jG2lCVTMbARz5f0JVbH7ZR VhWPWat1SmmDlfM8v_uhjWEwf7e_dbd_cNpO4phq2E458wSqAT1vZeYGoNMX.cP16GsvJanyr41A xUAQP1YnjwV8F0O5zc_uNBlWAF5YV0o0RvajIIj5yuff8UjUUqM4Wy3Cru1v5HJiYB5HYKbH1utI PYHNwg0Z9DD2W2sMjKjUTX_I1.laHk5Rqbh9WET3.ooP54E.Rtg6QfRjFWaHre3t5_1pgHLGDU5n pwK8Gu0Pr4EHyDFemtJF5ZujE_43MsP_6VIUN2tSU1MMB4nOCVqJK3obIKASD50M_.HxbjH8z_bG nGs5fKQaUhGNC081UlMmJ5sgviY.Wn.pd6Y_gHFnQ6pp1BG5cbILrehodw49FPOUq5DT93Gt6xGl LOfkp5DjruKly8..hlGx78bJ9vuwF2J9K9v3xUZ9mrZokF4Q0PmDy4F7buHetR50_6h5dV3.wJq9 a9whmKlqO_m6cepDFPHz_cxhDnM6pan3gb7Ui_8hhXNBGfcYhI3DGNBhxYmWUgTlbd1ymv5B3qJ. mCf3pcYmQUg637Z6vRVGP1mLEBmmvt4wyBwaQTjLxwykbDo0hdq.1z18j4AwV9sNJTk3_63pIoYz pGp2WIn4Eoze4evhkG5gw6sJTKMNOZPSqDM.xgwFPEL.RCCf_ZuzBliWPPcDtoV.sxbB_3gUrmmD _ieNug5z95C1H3g20YFmYE5kLeykJ5wAGhtiTcESbMt6uUyQRS_GBr4UUTdO83J.zxKo_D5LE5oj Iu7K8S4yULwD2xiNKZByE.xdPWa_Fnd9osI96B0D0VrwCOYFx_I3Jud88pwTo4NHgKN7yINDcG5N P74i_9WJ8_Y5ThOqamJGDuCCkyL3B8Hdk7.Mb_kmzVJCMZO0qrA-- X-Sonic-MF: X-Sonic-ID: 24645ac0-9401-4ded-ae18-83e97138d77a Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.ne1.yahoo.com with HTTP; Sat, 2 Mar 2024 15:06:29 +0000 Date: Sat, 2 Mar 2024 15:06:28 +0000 (UTC) From: Filippo Moretti To: Freebsd-stable List Message-ID: <1646351680.3606510.1709391988165@mail.yahoo.com> Subject: Error compiling xf86-video-intel List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_3606509_1912734134.1709391988164" References: <1646351680.3606510.1709391988165.ref@mail.yahoo.com> X-Mailer: WebService/1.1.22103 YMailNorrin X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.99)[-0.991]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[66.163.184.200:from]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[66.163.184.200:from]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+] X-Rspamd-Queue-Id: 4Tn7Zx5sJ8z3xnt ------=_Part_3606509_1912734134.1709391988164 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable -- intel_virtual_output-virtual.o --- virtual.c:499:11: error: use of undeclared identifier 'ETIME' =C2=A0 499 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return -ETIME; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 ^ virtual.c:506:11: error: use of undeclared identifier 'ETIME' =C2=A0 506 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return -ETIME; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 ^ virtual.c:3008:12: error: use of undeclared identifier 'ETIME' =C2=A03008 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 return -ETIME; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= ^ virtual.c:3042:16: warning: 'format' attribute argument not supported: gnu_= printf [-Wignored-attributes] =C2=A03042 | __attribute__((format(gnu_printf, 3, 4))) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ^ virtual.c:3056:40: warning: format string is not a string literal [-Wformat= -nonliteral] =C2=A03056 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 len =3D vsnpri= ntf(buf+4, sizeof(buf)-4, format, va)+5; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 ^~~~~~ virtual.c:3465:17: error: use of undeclared identifier 'ETIME' =C2=A03465 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (ret !=3D -ETIME)= { =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ^ 2 warnings and 4 errors generated. *** [intel_virtual_output-virtual.o] Error code 1 make[2]: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video= -intel-b74b67f0f321875492968f7097b9d6e82a66d7df/tools 1 error make[2]: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video= -intel-b74b67f0f321875492968f7097b9d6e82a66d7df/tools make[1]: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video= -intel-b74b67f0f321875492968f7097b9d6e82a66d7df make: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-in= tel-b74b67f0f321875492968f7097b9d6e82a66d7df =3D=3D=3D> Compilation failed unexpectedly. Try to set MAKE_JOBS_UNSAFE=3Dyes and rebuild before reporting the failure = to the maintainer. *** Error code 1 Stop. make[1]: stopped in /usr/ports/x11-drivers/xf86-video-intel *** Error code 1 Stop. make: stopped in /usr/ports/x11-drivers/xf86-video-intel [root@ROXY /usr/ports/x11-drivers/xf86-video-intel]#=20 This is the error message in my system :[root@ROXY /usr/ports/x11-drivers/x= f86-video-intel]# uname -a FreeBSD ROXY 14.0-STABLE FreeBSD 14.0-STABLE #22 stable/14-n266661-a727d8d7= f50f: Fri Feb=C2=A0 9 22:57:13 CET 2024=C2=A0=C2=A0=C2=A0=C2=A0 filippo@ROX= Y:/usr/obj/usr/src/amd64.amd64/sys/ROXY amd64 Filippo ------=_Part_3606509_1912734134.1709391988164 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
-- intel_virtual_output-virtual.o ---virtual.c:499:11: error: use of undeclared identifier 'ETIME'
  4= 99 |            = ;     return -ETIME;
      = |            &n= bsp;            ^virtual.c:506:11: error: use of undeclared identifier 'ETIME'
  50= 6 |            =      return -ETIME;
      |=             &nb= sp;            ^
= virtual.c:3008:12: error: use of undeclared identifier 'ETIME'
 300= 8 |            =              re= turn -ETIME;
      |    &nb= sp;            =             &nb= sp;   ^
virtual.c:3042:16: warning: 'format' attribute argumen= t not supported: gnu_printf [-Wignored-attributes]
 3042 | __attrib= ute__((format(gnu_printf, 3, 4)))
      | =             &nb= sp;  ^
virtual.c:3056:40: warning: format string is not a string li= teral [-Wformat-nonliteral]
 3056 |     &n= bsp;   len =3D vsnprintf(buf+4, sizeof(buf)-4, format, va)+5;
=       |       &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;  ^~~~~~
virtual.c:3465:17: error: use of undeclared identifier '= ETIME'
 3465 |         = ;            &n= bsp;           if (ret != =3D -ETIME) {
      |    &n= bsp;            = ;            &n= bsp;            = ;   ^
2 warnings and 4 errors generated.
*** [intel_virtual= _output-virtual.o] Error code 1

make[2]: stopped in /usr/ports/x11-d= rivers/xf86-video-intel/work/xf86-video-intel-b74b67f0f321875492968f7097b9d= 6e82a66d7df/tools
1 error

make[2]: stopped in /usr/ports/x11-driv= ers/xf86-video-intel/work/xf86-video-intel-b74b67f0f321875492968f7097b9d6e8= 2a66d7df/tools

make[1]: stopped in /usr/ports/x11-drivers/xf86-video= -intel/work/xf86-video-intel-b74b67f0f321875492968f7097b9d6e82a66d7df
make: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-= intel-b74b67f0f321875492968f7097b9d6e82a66d7df
=3D=3D=3D> Compilation= failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=3Dyes and rebuild befo= re reporting the failure to
the maintainer.
*** Error code 1

S= top.
make[1]: stopped in /usr/ports/x11-drivers/xf86-video-intel
*** = Error code 1

Stop.
make: stopped in /usr/ports/x11-drivers/xf86-v= ideo-intel
[root@ROXY /usr/ports/x11-drivers/xf86-video-intel]#

=
This is the error message in my system :
[root@ROX= Y /usr/ports/x11-drivers/xf86-video-intel]# uname -a
FreeBSD ROXY 14.0-S= TABLE FreeBSD 14.0-STABLE #22 stable/14-n266661-a727d8d7f50f: Fri Feb = 9 22:57:13 CET 2024     filippo@ROXY:/usr/obj/usr/src/= amd64.amd64/sys/ROXY amd64

Filippo
------=_Part_3606509_1912734134.1709391988164-- From nobody Sun Mar 3 04:28:20 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnTNG6M9Fz5BtDM for ; Sun, 3 Mar 2024 04:28:30 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnTNF5tDDz43L9 for ; Sun, 3 Mar 2024 04:28:29 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.17.1/8.17.1) with ESMTPS id 4234SLK6065562 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 2 Mar 2024 23:28:22 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.17.1/8.17.1/Submit) id 4234SLCr065561; Sat, 2 Mar 2024 23:28:21 -0500 (EST) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26083.64612.717082.366639@hergotha.csail.mit.edu> Date: Sat, 2 Mar 2024 23:28:20 -0500 From: Garrett Wollman To: Rick Macklem Cc: stable@freebsd.org Subject: Re: 13-stable NFS server hang In-Reply-To: References: <26078.50375.679881.64018@hergotha.csail.mit.edu> X-Mailer: VM 8.2.0b under 29.1 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Sat, 02 Mar 2024 23:28:22 -0500 (EST) X-Spam-Status: No, score=-4.5 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.90 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[wollman]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; TAGGED_RCPT(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4TnTNF5tDDz43L9 I wrote previously: > PID TID COMM TDNAME KSTACK > 997 108481 nfsd nfsd: master mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > 997 960918 nfsd nfsd: service mi_switch sleepq_timedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > 997 962232 nfsd nfsd: service mi_switch _cv_wait txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline I spent some time this evening looking at this last stack trace, and stumbled across the following comment in sys/contrib/openzfs/module/zfs/dmu.c: | /* | * Enable/disable forcing txg sync when dirty checking for holes with lseek(). | * By default this is enabled to ensure accurate hole reporting, it can result | * in a significant performance penalty for lseek(SEEK_HOLE) heavy workloads. | * Disabling this option will result in holes never being reported in dirty | * files which is always safe. | */ | int zfs_dmu_offset_next_sync = 1; I believe this explains why vn_copy_file_range sometimes takes much longer than a second: our servers often have lots of data waiting to be written to disk, and if the file being copied was recently modified (and so is dirty), this might take several seconds. I've set vfs.zfs.dmu_offset_next_sync=0 on the server that was hurting the most and am watching to see if we have more freezes. If this does the trick, then I can delay deploying a new kernel until April, after my upcoming vacation. -GAWollman From nobody Sun Mar 3 05:25:18 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnVds60c9z5C0P2 for ; Sun, 3 Mar 2024 05:25:21 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "garrett.wollman.name", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnVds0dHLz499Q for ; Sun, 3 Mar 2024 05:25:21 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bimajority.org (policy=none); spf=pass (mx1.freebsd.org: domain of wollman@hergotha.csail.mit.edu designates 2001:470:1f06:ccb::2 as permitted sender) smtp.mailfrom=wollman@hergotha.csail.mit.edu Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.17.1/8.17.1) with ESMTPS id 4235PJH0066202 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 3 Mar 2024 00:25:19 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.17.1/8.17.1/Submit) id 4235PJFM066201; Sun, 3 Mar 2024 00:25:19 -0500 (EST) (envelope-from wollman) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <26084.2494.962383.278446@hergotha.csail.mit.edu> Date: Sun, 3 Mar 2024 00:25:18 -0500 From: Garrett Wollman To: stable@freebsd.org Cc: Rick Macklem Subject: Re: 13-stable NFS server hang In-Reply-To: <26083.64612.717082.366639@hergotha.csail.mit.edu> References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> X-Mailer: VM 8.2.0b under 29.1 (amd64-portbld-freebsd13.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (hergotha.csail.mit.edu [0.0.0.0]); Sun, 03 Mar 2024 00:25:19 -0500 (EST) X-Spam-Status: No, score=-4.5 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS,NICE_REPLY_A,T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on hergotha.csail.mit.edu X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.68 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; NEURAL_HAM_MEDIUM(-0.78)[-0.783]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f06:ccb::2]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[wollman]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TAGGED_RCPT(0.00)[]; FREEMAIL_CC(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4TnVds0dHLz499Q < I believe this explains why vn_copy_file_range sometimes takes much > longer than a second: our servers often have lots of data waiting to > be written to disk, and if the file being copied was recently modified > (and so is dirty), this might take several seconds. I've set > vfs.zfs.dmu_offset_next_sync=0 on the server that was hurting the most > and am watching to see if we have more freezes. In case anyone is wondering why this is an issue, it's the combination of two factors: 1) vn_generic_copy_file_range() attempts to preserve holes in the source file. 2) ZFS does automatic hole-punching on write for filesystems where compression is enabled. It happens in the same code path as compression, checksum generation, and redundant-write suppression, and thus does not happen until the dirty blocks are about to be committed to disk. So if the file is dirty, ZFS doesn't "know" whether thare where the then-extant holes are until a sync has completed. While vn_generic_copy_file_range() has a flag to stop and return partial success after a second of copying, this flag does not affect sleeps internal to the filesystem, so zfs_holey() can sleep indefinitely and vn_generic_copy_file_range() can't do anything about it until the sync has already happened. -GAWollman From nobody Sun Mar 3 15:37:59 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnmF53ZP4z5D0xq for ; Sun, 3 Mar 2024 15:38:17 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-yb1-xb2b.google.com (mail-yb1-xb2b.google.com [IPv6:2607:f8b0:4864:20::b2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnmF45Jc4z4M1v for ; Sun, 3 Mar 2024 15:38:16 +0000 (UTC) (envelope-from kob6558@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=mc2P46B6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kob6558@gmail.com designates 2607:f8b0:4864:20::b2b as permitted sender) smtp.mailfrom=kob6558@gmail.com Received: by mail-yb1-xb2b.google.com with SMTP id 3f1490d57ef6-dc6d9a8815fso3802861276.3 for ; Sun, 03 Mar 2024 07:38:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709480295; x=1710085095; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ERJ9YedjL2T2FS0tQ3IdLlkEUXaQlCh86EgBmwAUaAg=; b=mc2P46B6tEKwTWMF8eHL1vQx0Pao7kZvb2pcTS0aEVgW4acJtmiX+GLP6ns73Vvlj1 m57tW7Npg5/HsSkcvqMrwehDUMc9naq3e2FWqIfUYCIXPs5+ee6WRC1NuMWCUZeGY5Np KwZvzrdv1SVJ3gZL9PIQLc5zRoRIKwDvbERaBY/g4+N7omuXzSJYZ1ZqCkBPQ0TuXEez UVmKrB/crRbQSASlBAWqyUwYjG5mQz4Nm69qTETWpmZ4YTMpXzV9TptiOYAdxbj3O2mo e982C1zfnc7/QK+JT/glTrfiA/4O5g2nPlNb4sUMEFWxaaneEgUkM5/A9l8xtJ8VCKu4 TXPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709480295; x=1710085095; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ERJ9YedjL2T2FS0tQ3IdLlkEUXaQlCh86EgBmwAUaAg=; b=kydNy/2+mVOEtsARdH+NPcTqCCjnC0teCQwTsH542PSD8qxSLnXVBU6Ixfzz1B4xV1 c+zGwC3bkALOI0qpMR2/NedagtBKChc7kDyndYnhsqj79fu6skV7IGVMX/TN0O5jl1DF +QC5O+1sXYVeDYW/aF5iKBHZfDhZnWTdcV6wJ56XMvW6ch8XIie1MqnnnWKpZUeoRONg t1ZIgMJ4Wlxwo81sYSppUa1J6tnKyYHKa4EvfUd57Frdb0Lk4EL9hD1uDmCF6EPMrn4i esEOpoTnH/4tOCS7+ZtLAIoCN5nYwETVWpPxmlnbcR6kPIYw4dZP4njacAmM9ncqZ9nB 2W8w== X-Gm-Message-State: AOJu0Yyq1ULLidv00OSIk73ZYHxkWy1nGCWdz3YLCSl8Ixkhu2aOQgfE IqgB5z0eKIIZzxX1xhPhBzfIDGWKyuJ4UiT87mYKKm74v1sNlawMDUGQybSnxxdLIwZ/xBA2mlV yDJr8XvVlKrjBkjs7wfUrN89wsHRK1jXU6W8= X-Google-Smtp-Source: AGHT+IFPhHVmyvAHVf2lj5TCEj9QDUcNl48ouZuCmDDQ3kIWRC/OEiy68WFzRnu9y0h/PA/hCooQANIxPnWi5eOUCkE= X-Received: by 2002:a25:e807:0:b0:dcd:5f08:3666 with SMTP id k7-20020a25e807000000b00dcd5f083666mr5211727ybd.29.1709480295586; Sun, 03 Mar 2024 07:38:15 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <1646351680.3606510.1709391988165.ref@mail.yahoo.com> <1646351680.3606510.1709391988165@mail.yahoo.com> In-Reply-To: <1646351680.3606510.1709391988165@mail.yahoo.com> From: Kevin Oberman Date: Sun, 3 Mar 2024 07:37:59 -0800 Message-ID: Subject: Re: Error compiling xf86-video-intel To: Filippo Moretti Cc: Freebsd-stable List Content-Type: multipart/alternative; boundary="000000000000c4e2ab0612c36552" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.69 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FORGED_SENDER(0.30)[rkoberman@gmail.com,kob6558@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[rkoberman@gmail.com,kob6558@gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b2b:from] X-Rspamd-Queue-Id: 4TnmF45Jc4z4M1v --000000000000c4e2ab0612c36552 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Mar 2, 2024 at 7:06=E2=80=AFAM Filippo Moretti wrote: > -- intel_virtual_output-virtual.o --- > virtual.c:499:11: error: use of undeclared identifier 'ETIME' > 499 | return -ETIME; > | ^ > virtual.c:506:11: error: use of undeclared identifier 'ETIME' > 506 | return -ETIME; > | ^ > virtual.c:3008:12: error: use of undeclared identifier 'ETIME' > 3008 | return -ETIME; > | ^ > virtual.c:3042:16: warning: 'format' attribute argument not supported: > gnu_printf [-Wignored-attributes] > 3042 | __attribute__((format(gnu_printf, 3, 4))) > | ^ > virtual.c:3056:40: warning: format string is not a string literal > [-Wformat-nonliteral] > 3056 | len =3D vsnprintf(buf+4, sizeof(buf)-4, format, va)+5; > | ^~~~~~ > virtual.c:3465:17: error: use of undeclared identifier 'ETIME' > 3465 | if (ret !=3D -ETIME) { > | ^ > 2 warnings and 4 errors generated. > *** [intel_virtual_output-virtual.o] Error code 1 > > make[2]: stopped in > /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-b74b67f0f32= 1875492968f7097b9d6e82a66d7df/tools > 1 error > > make[2]: stopped in > /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-b74b67f0f32= 1875492968f7097b9d6e82a66d7df/tools > > make[1]: stopped in > /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-b74b67f0f32= 1875492968f7097b9d6e82a66d7df > > make: stopped in > /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-b74b67f0f32= 1875492968f7097b9d6e82a66d7df > =3D=3D=3D> Compilation failed unexpectedly. > Try to set MAKE_JOBS_UNSAFE=3Dyes and rebuild before reporting the failur= e to > the maintainer. > *** Error code 1 > > Stop. > make[1]: stopped in /usr/ports/x11-drivers/xf86-video-intel > *** Error code 1 > > Stop. > make: stopped in /usr/ports/x11-drivers/xf86-video-intel > [root@ROXY /usr/ports/x11-drivers/xf86-video-intel]# > > > This is the error message in my system : > [root@ROXY /usr/ports/x11-drivers/xf86-video-intel]# uname -a > FreeBSD ROXY 14.0-STABLE FreeBSD 14.0-STABLE #22 > stable/14-n266661-a727d8d7f50f: Fri Feb 9 22:57:13 CET 2024 > filippo@ROXY:/usr/obj/usr/src/amd64.amd64/sys/ROXY amd64 > > Filippo > People keep reporting problems with xf86-video-intel when it really should not be used on any modern system. Intel dropped support for this driver years ago. Unless you have a really old system with a device that predates at least Sandy Bridge, the proper thing to do is to simply delete it. I'm not sure why xorg still prioritizes it ahead of modesetting which is the correct driver for any modern system using Intel graphics. I would like to see some information that modesetting is the right driver for any recent Intel graphics, but it seems obvious that the old Intel driver is needed for Intel graphics, even though it is the wrong answer. --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000c4e2ab0612c36552 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Sat, Mar 2, 2024 at 7:06=E2= =80=AFAM Filippo Moretti <filip= pomore@yahoo.com> wrote:
-- intel_virtual_output-virtual.o ---
virtual.c:499:11: error: use= of undeclared identifier 'ETIME'
=C2=A0 499 |=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 return -ETIME;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ^
virtual.c:506:11: err= or: use of undeclared identifier 'ETIME'
=C2=A0 506 |=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 return -ETIME;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ^
virtual.c:30= 08:12: error: use of undeclared identifier 'ETIME'
=C2=A03008 |= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 retur= n -ETIME;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 ^
virtual.c:3042:16: warning: 'format' attribute ar= gument not supported: gnu_printf [-Wignored-attributes]
=C2=A03042 | __a= ttribute__((format(gnu_printf, 3, 4)))
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 ^
virtual.c:3056:40: warning: format string is not a str= ing literal [-Wformat-nonliteral]
=C2=A03056 |=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 len =3D vsnprintf(buf+4, sizeof(buf)-4, format, va= )+5;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 ^~~~~~
virtual.c:3465:17: error: use of undeclared id= entifier 'ETIME'
=C2=A03465 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 if (ret !=3D -ETIME) {
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ^
2 warnings and 4 errors generated.*** [intel_virtual_output-virtual.o] Error code 1

make[2]: stopped= in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-b74b67f0f= 321875492968f7097b9d6e82a66d7df/tools
1 error

make[2]: stopped in= /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-b74b67f0f321= 875492968f7097b9d6e82a66d7df/tools

make[1]: stopped in /usr/ports/x1= 1-drivers/xf86-video-intel/work/xf86-video-intel-b74b67f0f321875492968f7097= b9d6e82a66d7df

make: stopped in /usr/ports/x11-drivers/xf86-video-in= tel/work/xf86-video-intel-b74b67f0f321875492968f7097b9d6e82a66d7df
=3D= =3D=3D> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE= =3Dyes and rebuild before reporting the failure to
the maintainer.
**= * Error code 1

Stop.
make[1]: stopped in /usr/ports/x11-drivers/x= f86-video-intel
*** Error code 1

Stop.
make: stopped in /usr/p= orts/x11-drivers/xf86-video-intel
[root@ROXY /usr/ports/x11-drivers/xf86= -video-intel]#


This is the error message in my system :
[root@ROXY /usr/ports/x11-drivers/xf86-video-intel]# uname -a=
FreeBSD ROXY 14.0-STABLE FreeBSD 14.0-STABLE #22 stable/14-n266661-a727= d8d7f50f: Fri Feb=C2=A0 9 22:57:13 CET 2024=C2=A0=C2=A0=C2=A0=C2=A0 filippo= @ROXY:/usr/obj/usr/src/amd64.amd64/sys/ROXY amd64

Filippo
=C2=A0
People keep reporting problems with xf86-video-intel whe= n it really should not be used on any modern system. Intel dropped support = for this driver years ago. Unless you have a really old system with a devic= e that predates at least Sandy Bridge, the proper thing to do is to simply = delete it. I'm not sure why xorg still prioritizes it ahead of modesett= ing which is the correct driver for any modern system using Intel graphics.=

I would like to see some information th= at modesetting is the right driver for any recent Intel graphics, but it se= ems obvious that the old Intel driver is needed for Intel graphics, even th= ough it is the wrong answer.
--
Kevin Ob= erman, Part time kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
--000000000000c4e2ab0612c36552-- From nobody Sun Mar 3 16:25:57 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnnJC2qdGz5BrFc for ; Sun, 3 Mar 2024 16:26:03 +0000 (UTC) (envelope-from filippomore@yahoo.com) Received: from sonic312-24.consmr.mail.ne1.yahoo.com (sonic312-24.consmr.mail.ne1.yahoo.com [66.163.191.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnnJB4qlWz4R5W for ; Sun, 3 Mar 2024 16:26:02 +0000 (UTC) (envelope-from filippomore@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709483159; bh=hkYN3RcTNLMzpTyMfqTAFBmM37g24RRP+wb5CvqPDH8=; h=Date:From:To:Cc:In-Reply-To:References:Subject:From:Subject:Reply-To; b=tK0wFfZnhkDe4mpW8txsHrcHxTW87p+hF+ofZzA/DBmbNyb+nIFHFubk0De6J47wBlxl3p2iOwQWP36TiTlyeFdsB8Icui++YlpnmtVeupRhHxK1i9tKiMYa4bfr7NJl9b1ChtF1tLIC01k9tiWGH5hwqGIrBtilafgy+6LjRjENwYZ1GKMG2pLeexNhLofX1F8+k0GNAFPfOEVrdin20TRXVTs41KqNpR9swq19Cf+brOJPy568i+tobtr6KDjUmlfyue8rJ+fB/Cacp0mHOrP/qxqndxD0akmVmjCcE0HosybVlc2ugc+TqDtskyXOhNtGOf1xswlsmrinM/fMCA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1709483159; bh=Kvh6N+q47EkXrzU6eGmJK2BTgsq5S3Au0f4vg6DKGdN=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=rGclKtLr2mHkzA4Re0wMKaVFkvvknWxqqfpFP7EcSeEV2bE3iGsXf1eZCRChccYAcMyAWYWfy02t+QEK+Sy23+SmiQIoE40kLvb/GQJVArnXtkbYpP4Om4jEGiml8latOLCX0s9i6zzcs7Iw2kKeRTU3A7OKcXkNJlFRUks4KEvG9zUrG9tGviLwXSdfmhcURKE+GAscNnyZniu8Aeihm2NAj1LPRF/b8mbOaLkqqzJKpP5HNsfWiXsCZPYGVnnV3aRVZXZeDw11hoTfanTJFjcH08CKZYcDDlNcwtlDIOz3EOc62rBUq9ebNGBdwt93LYpOMSTJpErHmEyHIYMC7w== X-YMail-OSG: Yc3yV9sVM1l684B8DBYco1FgU3Q_l7r0N_JBVozoL9qa48fuw65D7vAICw.lb23 KIhDkz2ZH9ab8PnkiEOzEj3TZ6s6kvKeSS.aceRxo9iEFmpmCXK5iU8QjHp1AqTE1LFGrdx4iYyW dngYnWahrj2Du72.NfrWf1DpaZJsGdR5.qKOT7AxZ4zFMVNp6WK8gcP0npKbF5Fw6.yARZE9kfd9 AEDD7iYPyHS.ARIXeAhr9BdjsB7.qc3gPEjAuqu840FYl.Onwy70WoIU4.TEFa20WNB.3xjv0fyh Xrzi74RBgb09ZvU1wY.m6kjSss_518W9rBFX2SNURHHbArTMuwxKXT8ru0yEHORGLazcTvUx_mRY cvYDqhpw0tRNNGSnw8wx0e581OeYRe6plSN.hFVKdOCukn_Md1BZ2l2JlD_NEd8VPV.gLQ8kJAw7 v0Nfoo6ZY6h5g..tG2AgaRVvLxbsAp9WyUGdKGrPHGeLvXQ1YVeobFi_aeyQjgxj4vlVmU4TxxMj 4UYiwWwuDmRJshrAW0d_xXnNouJOXf7UyFDcByeWgyvVkS0lHP.SS1RC2oiGvnYbzyXseF.lZzub a6vJF9C0PxsRG3reMKk91KASp8H0hjxRP41FlmQPbiGmu.Sqv2yRiKA2HQtMn4Etg6gDwSjLb9o3 Jnv8PvL2GUiRXyjX6tlV0Hu86cn8gmJKAh.q5xG6b.z3Mhcqc5zsFyuagsCYcFJ.TMrZ.U4ZXt5d oI8BQs1FXjbMimnfv0WE.haNWRFbQBQCuJnN22nsKAy9zdv0d01DRo8jDvm_16GQc.M7bRDr0LB5 wP17ciFW4.tU9IHZ1Bbb9mspgSU_8wFVbKGgJwnuqcEHkUu6jYYx6L0.2QtrolH7msnHTRmtVHk0 KEhL781f0YHkCstDop79_yVbW2Mk3tlLSunogJsCXMBCElsQLeyteBt3RIG0xS7tHImeKSIFH4sJ APe0eNB8jt1fpHNCKukPz0C68qPCQG5zHDI9bRvwQB2yGGjDiZeOvbnGykLLKb2p7ViqIQdwZwRZ PCqHslXNtDezckGPQFCXQUoj90yu_7rmc3VDHvlueQHEICL1CrpAusBNftTeSTn7bLF6YmDhRF97 CsGFdJo_s_bI8IXarnd3yGl9M0fL.O1QRiEypuU6j1g0vo.NSc.G6AMOPO_3eW5Zo.Df.0gHWs8k pbDoVo0fFlvHOtuFqN4v8oRGyLJpYLeRabfEtTJv4SsHxG_La4lEZqY.4vGetBG6w12ClTzpEM0N _ZqBDLgZekxCBqbkx5Vh844yjUOCWLSNQLkwlgDZxUEdUqAFQdeGWhU6qzOMRgf7c_e7T953o9ZR rm0rilCpPK6fXKl2kshWuLyvjN2ndcRTsCM5Rle8y65IQsDmiS3mrgZcZAlTebt7ZKLAFgZwmDgt VXBujAD_4q5bAEq4O2yrgBSDR_0wwZeyjAnb75ulsBusBSe7hajHu0tOMdfQqP_iE_3CfpBF4.Qr 6RaDFKq_S7RChRdaL073324gpsxh6.XSDrspkxt_l_u99zIOM_PwmY4MwsNhVUtcaYRClrU.m0cj uLVxvGrz_DP_CU4i.GCra3DFm4GvjgYWKVY9PcMLsTjt78r__8krgBzNeSU2bNmd7cYXKCgTJ4AW BZY1AHjM4TkD4_dSLrYvvT9tqc6i78Csy.BtZZKkpHnexbMGSF7nZGa7F3AGyIvl_twBv34d0H1i _JkVH0zK_lGq4xcMI0syRU58LE.CJ3i2kNXtddxwSiRNCHBGPwEix8jfV20K4fcTbBmsiVQIqZZY 0MS5xnTCmv909iIGzT6mvfPvtHOVDtYlRw8OW0PGYDcJmqp1bIE5SfPZ.cCHf152WHVqiKrmxbrw gSdyrUFKU6oHfypb0zOsNWVRljLCSzUkjp02JWcJbWx5AeMkWnszWg9Rze07dHGSQfU1yZAyg.cn B3XiBVJoGhTaOMV36XFHZJafrH9_mz8BDZn3Zosu7d7SMz0J6yGMOHWxJd_vQNXy29xanfuJDWrV XTJKyHftOwCnA9n8m357nCZH8Opk3NVx8VRSwQrj.BZSq6mbVi5pNvn5vcxK9FtPYKHL0vpDcWFk ofBt4pXACk9YdmpYzEPLdgPGfAPXXzi1OaHuRrdqDAd8NfKXO5HiEQE1tk2ZnEZDEwfx6sfoPezx l_wkB3vEsKNnpDsmU20Dq971ANoap2kVlZp.suAQHG1KG5GohgtOwBJgUrRBMV5G8PI6gcxSWbX9 tAfvZsMowYA-- X-Sonic-MF: X-Sonic-ID: fd20aa23-1b3f-4aa9-8520-9a6f1a86f3b1 Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.ne1.yahoo.com with HTTP; Sun, 3 Mar 2024 16:25:59 +0000 Date: Sun, 3 Mar 2024 16:25:57 +0000 (UTC) From: Filippo Moretti To: Kevin Oberman Cc: Freebsd-stable List Message-ID: <476385005.3877519.1709483157919@mail.yahoo.com> In-Reply-To: References: <1646351680.3606510.1709391988165.ref@mail.yahoo.com> <1646351680.3606510.1709391988165@mail.yahoo.com> Subject: Re: Error compiling xf86-video-intel List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_3877518_1248211290.1709483157917" X-Mailer: WebService/1.1.22103 YMailNorrin X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US] X-Rspamd-Queue-Id: 4TnnJB4qlWz4R5W ------=_Part_3877518_1248211290.1709483157917 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thank you for your explanation,my computer is an old core duo but X is wor= king anyway even with my older build.SincerelyFilippo On Sunday, March 3, 2024 at 04:38:32 PM GMT+1, Kevin Oberman wrote: =20 =20 On Sat, Mar 2, 2024 at 7:06=E2=80=AFAM Filippo Moretti wrote: -- intel_virtual_output-virtual.o --- virtual.c:499:11: error: use of undeclared identifier 'ETIME' =C2=A0 499 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return -ETIME; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 ^ virtual.c:506:11: error: use of undeclared identifier 'ETIME' =C2=A0 506 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return -ETIME; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 ^ virtual.c:3008:12: error: use of undeclared identifier 'ETIME' =C2=A03008 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 return -ETIME; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= ^ virtual.c:3042:16: warning: 'format' attribute argument not supported: gnu_= printf [-Wignored-attributes] =C2=A03042 | __attribute__((format(gnu_printf, 3, 4))) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ^ virtual.c:3056:40: warning: format string is not a string literal [-Wformat= -nonliteral] =C2=A03056 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 len =3D vsnpri= ntf(buf+4, sizeof(buf)-4, format, va)+5; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 ^~~~~~ virtual.c:3465:17: error: use of undeclared identifier 'ETIME' =C2=A03465 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (ret !=3D -ETIME)= { =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ^ 2 warnings and 4 errors generated. *** [intel_virtual_output-virtual.o] Error code 1 make[2]: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video= -intel-b74b67f0f321875492968f7097b9d6e82a66d7df/tools 1 error make[2]: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video= -intel-b74b67f0f321875492968f7097b9d6e82a66d7df/tools make[1]: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video= -intel-b74b67f0f321875492968f7097b9d6e82a66d7df make: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-in= tel-b74b67f0f321875492968f7097b9d6e82a66d7df =3D=3D=3D> Compilation failed unexpectedly. Try to set MAKE_JOBS_UNSAFE=3Dyes and rebuild before reporting the failure = to the maintainer. *** Error code 1 Stop. make[1]: stopped in /usr/ports/x11-drivers/xf86-video-intel *** Error code 1 Stop. make: stopped in /usr/ports/x11-drivers/xf86-video-intel [root@ROXY /usr/ports/x11-drivers/xf86-video-intel]#=20 This is the error message in my system :[root@ROXY /usr/ports/x11-drivers/x= f86-video-intel]# uname -a FreeBSD ROXY 14.0-STABLE FreeBSD 14.0-STABLE #22 stable/14-n266661-a727d8d7= f50f: Fri Feb=C2=A0 9 22:57:13 CET 2024=C2=A0=C2=A0=C2=A0=C2=A0 filippo@ROX= Y:/usr/obj/usr/src/amd64.amd64/sys/ROXY amd64 Filippo =C2=A0People keep reporting problems with xf86-video-intel when it really s= hould not be used on any modern system. Intel dropped support for this driv= er years ago. Unless you have a really old system with a device that predat= es at least Sandy Bridge, the proper thing to do is to simply delete it. I'= m not sure why xorg still prioritizes it ahead of modesetting which is the = correct driver for any modern system using Intel graphics. I would like to see some information that modesetting is the right driver f= or any recent Intel graphics, but it seems obvious that the old Intel drive= r is needed for Intel graphics, even though it is the wrong answer. --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 =20 ------=_Part_3877518_1248211290.1709483157917 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thank you for your explanati= on,my computer is an old core duo but X is working anyway even with my olde= r build.
Sincerely
Filippo

=20
=20
On Sunday, March 3, 2024 at 04:38:32 PM GMT+1, Kevin Ob= erman <rkoberman@gmail.com> wrote:


On Sat, Mar 2, 2024 at 7:06=E2=80=AFAM Fi= lippo Moretti <filippomore@yahoo.com> wrote:
-- intel_virt= ual_output-virtual.o ---
virtual.c:499:11: error: use of = undeclared identifier 'ETIME'
  499 |  &nb= sp;            =   return -ETIME;
      |&nb= sp;            =             ^
virtual.c:506:11: error: use of undeclared identifier 'ETIME'  506 |       &nbs= p;         return -ETIME;
      |     &nb= sp;            =        ^
virtual.c:3008:12:= error: use of undeclared identifier 'ETIME'
 3008 |=             &nb= sp;            retur= n -ETIME;
      |  &nb= sp;            =             &nb= sp;     ^
virtual.c:3042:16: warning:= 'format' attribute argument not supported: gnu_printf [-Wignored-attribute= s]
 3042 | __attribute__((format(gnu_printf, 3, 4)))=
      |    =             ^
virtual.c:3056:40: warning: format string is not a string liter= al [-Wformat-nonliteral]
 3056 |   &n= bsp;     len =3D vsnprintf(buf+4, sizeof(buf)-4, format= , va)+5;
      |  &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;      ^~~~~~
virtual.c:3465= :17: error: use of undeclared identifier 'ETIME'
 34= 65 |            = ;            &n= bsp;        if (ret !=3D -ETIME) {
      |     =             &nb= sp;            =             &nb= sp;  ^
2 warnings and 4 errors generated.
*** [intel_virtual_output-virtual.o] Error code 1

make[2]: stopped in /usr/ports/x11-drivers/xf86-video= -intel/work/xf86-video-intel-b74b67f0f321875492968f7097b9d6e82a66d7df/tools=
1 error

make[2]: st= opped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-video-intel-b74b= 67f0f321875492968f7097b9d6e82a66d7df/tools

make[1]: stopped in /usr/ports/x11-drivers/xf86-video-intel/work/xf86-= video-intel-b74b67f0f321875492968f7097b9d6e82a66d7df

make: stopped in /usr/ports/x11-drivers/xf86-video-intel/wor= k/xf86-video-intel-b74b67f0f321875492968f7097b9d6e82a66d7df
=3D=3D=3D> Compilation failed unexpectedly.
Try to = set MAKE_JOBS_UNSAFE=3Dyes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make[1]: stopped in /u= sr/ports/x11-drivers/xf86-video-intel
*** Error code 1
Stop.
make: stopped in= /usr/ports/x11-drivers/xf86-video-intel
[root@ROXY /usr/= ports/x11-drivers/xf86-video-intel]#

=
This is the error message in my system :
[root@ROXY /usr/ports/x11-drivers/xf86-video-intel]# uname -a
FreeBSD ROXY 14.0-STABLE FreeBSD 14.0-STABLE #22 stable/14-n2666= 61-a727d8d7f50f: Fri Feb  9 22:57:13 CET 2024     = filippo@ROXY:/usr/obj/usr/src/amd64.amd64/sys/ROXY amd64

Filippo
 
People keep reporting problems with xf86-video-intel when it really should= not be used on any modern system. Intel dropped support for this driver ye= ars ago. Unless you have a really old system with a device that predates at= least Sandy Bridge, the proper thing to do is to simply delete it. I'm not= sure why xorg still prioritizes it ahead of modesetting which is the corre= ct driver for any modern system using Intel graphics.

I would like to = see some information that modesetting is the right driver for any recent In= tel graphics, but it seems obvious that the old Intel driver is needed for = Intel graphics, even though it is the wrong answer.
--
Kevi= n Oberman, Part time kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com
PGP Fingerp= rint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
------=_Part_3877518_1248211290.1709483157917-- From nobody Sun Mar 3 17:11:48 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnpKC1GqJz5Bw16 for ; Sun, 3 Mar 2024 17:11:59 +0000 (UTC) (envelope-from jo@bruelltuete.com) Received: from email.jo-t.de (seppel.jo-t.de [45.132.244.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnpKB17FNz4X9X for ; Sun, 3 Mar 2024 17:11:58 +0000 (UTC) (envelope-from jo@bruelltuete.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bruelltuete.com header.s=bruelltuete18a header.b=QD9oOLRH; dmarc=pass (policy=none) header.from=bruelltuete.com; spf=pass (mx1.freebsd.org: domain of jo@bruelltuete.com designates 45.132.244.126 as permitted sender) smtp.mailfrom=jo@bruelltuete.com DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bruelltuete.com; s=bruelltuete18a; t=1709485910; bh=jv9PjLAvtT8sjRZLWpJBxAmwTUf5AnIwixDbbT8yOBw=; h=Message-ID:Date:MIME-Version:To:From:Subject:From; b=QD9oOLRHjubDRXv29Vn1QOvhRBceGTNOQUx9K0Z3jddqK+qZe96n3GvdeoQdKEMMo Z52oTDTk4V2TdZjgkUZCLx7VLV5O4c6a0FKmGQsO6vHYpp1mSkm8c1y8whwFjZPhry VQYGvNtUdMiqlO6GRmLPX2XJ8s/VCgeSXsv1O2bIb4ufqHpFzYM4cUUCYudn8MrtKn jKXXZ4n7cjtavlVFpwfSyItgpuL18x0tANOq0uKn2S+bYKJbWLB4JOesT3iXMyL2/d Pvr9+jl7BwwprgjQTmtKSPbYnVCnONLHnzcUQCin587YHnqCauGx1eQBaRG9NuUbiz 7PllMF/KFrVig== Message-ID: <9d1a95e5-2972-4e52-88ef-8d7993275f14@bruelltuete.com> Date: Sun, 3 Mar 2024 17:11:48 +0000 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Language: en-GB To: FreeBSD Stable From: Johannes Totz Subject: Failed compiling releng/13.2, commit missing? Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: -- X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[bruelltuete.com,none]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[bruelltuete.com:s=bruelltuete18a]; MIME_GOOD(-0.10)[text/plain]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:197540, ipnet:45.132.244.0/22, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bruelltuete.com:+] X-Rspamd-Queue-Id: 4TnpKB17FNz4X9X Hi everyone, I tried compiling releng/13.2 on a fresh stable/13. That fails with: --- _bootstrap-tools-usr.sbin/zic --- /scratch/20240303_162512/stage/usr/src/contrib/tzcode/zic.c:464:8: error: an attribute list cannot appear here 464 | static ATTRIBUTE_NORETURN void | ^~~~~~~~~~~~~~~~~~ /scratch/20240303_162512/stage/usr/src/contrib/tzcode/private.h:471:30: note: expanded from macro 'ATTRIBUTE_NORETURN' 471 | # define ATTRIBUTE_NORETURN [[noreturn]] | ^~~~~~~~~~~~ /scratch/20240303_162512/stage/usr/src/contrib/tzcode/zic.c:471:8: error: an attribute list cannot appear here 471 | static ATTRIBUTE_NORETURN void | ^~~~~~~~~~~~~~~~~~ /scratch/20240303_162512/stage/usr/src/contrib/tzcode/private.h:471:30: note: expanded from macro 'ATTRIBUTE_NORETURN' 471 | # define ATTRIBUTE_NORETURN [[noreturn]] | ^~~~~~~~~~~~ /scratch/20240303_162512/stage/usr/src/contrib/tzcode/zic.c:669:8: error: an attribute list cannot appear here 669 | static ATTRIBUTE_NORETURN void | ^~~~~~~~~~~~~~~~~~ /scratch/20240303_162512/stage/usr/src/contrib/tzcode/private.h:471:30: note: expanded from macro 'ATTRIBUTE_NORETURN' 471 | # define ATTRIBUTE_NORETURN [[noreturn]] | ^~~~~~~~~~~~ /scratch/20240303_162512/stage/usr/src/contrib/tzcode/zic.c:3778:8: error: an attribute list cannot appear here 3778 | static ATTRIBUTE_NORETURN void | ^~~~~~~~~~~~~~~~~~ /scratch/20240303_162512/stage/usr/src/contrib/tzcode/private.h:471:30: note: expanded from macro 'ATTRIBUTE_NORETURN' 471 | # define ATTRIBUTE_NORETURN [[noreturn]] | ^~~~~~~~~~~~ 4 errors generated. *** [zic.o] Error code 1 make[3]: stopped in /scratch/20240303_162512/stage/usr/src/usr.sbin/zic Compiling stable/13 itself is fine. Looking at the history of zic, comparing releng/13.2 and stable/13, https://github.com/freebsd/freebsd-src/commits/releng/13.2/contrib/tzcode/zic.c and https://github.com/freebsd/freebsd-src/commits/stable/13/contrib/tzcode/zic.c, we see that stable/13 has an extra commit "Update tzcode to 2023c." that does change that attribute/keyword order. Releng/13.2 has "import tzdata 2023d" and "...2024a", but no "...2023c". Is that commit simply missing on that branch? cheers, Johannes From nobody Sun Mar 3 21:14:44 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnvjY5Yd4z5CKwF for ; Sun, 3 Mar 2024 21:14:57 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnvjY1rCsz4DXs for ; Sun, 3 Mar 2024 21:14:57 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1031.google.com with SMTP id 98e67ed59e1d1-29a74c88f74so2463539a91.3 for ; Sun, 03 Mar 2024 13:14:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709500496; x=1710105296; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=nNvtF9/dyHz4G818mKjFJOLfH3CMn85UIbHGtp30V8Q=; b=MhxHfQ+5A+dn6sKQarOs+/GDkCKGPSQXo8hTKbdVqlaUre+6Ommn9GDzFUOfbMWE0t K1dtB1L0My4Ex1GQk0YdDburagWD+EvU0Wx44dFZgHEtw8DHxb1UEpq9AWsuQB7iUAMu TvMBelIggXNyyNB7EIJGSonfSy6St5bl76bDiZk9vM3HXlD85uLLFtrwoWUrIsfNkPS7 kJn/763hiHBChZyrvTQv9tY6ggNJ4LMGGXEhsh7dOPIwg5YXgf29bmYKm01jR1EbpwEW MJH8pjilhiQcm6LTSz/zXX87mbQ2MR5oWHxDsfN42chZawG/1icT6WenLHjddUAkXQLw cEUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709500496; x=1710105296; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nNvtF9/dyHz4G818mKjFJOLfH3CMn85UIbHGtp30V8Q=; b=Fo+tn5z+a4NjCxfecZdjsdCyFv+V4yphPLLAB7jcntE3yEolk1elOjyXsvgB394ASo sLCG2VXNMkqal+IG0TaD/2M1goyMnf2CdB0Nx9hpovpVzVpsNVA1MkADU3rBAyREIWAV 3D5GjWj/OrJ2BsUjZ7pFOPv4bxY9GnAPD1s5ZbqTa6f2/dUiPw2lfgbUoqhO8kFA4XEm Q+zJLWexOhYCd32FTYjjDdKg6CfcdEJe13herzGxZ1BVWbV8pIOIcLQcfOzoj4Wmzg2Q Ze0PS4XbUB6MqnzJoxNBVvyGOE7M0eKeQYBGckfHIOZdkCjaaF8RXVAofBWvdwq2ywX0 9nIA== X-Gm-Message-State: AOJu0Yzlpl6IUXI1A6asDv+fPNGyvFIstdMpTPNy1F5sO95IjnUThSel YCmKDphtQ+3ire9duD8SZl/sTLGki0xSgXayYxMhpfCgKG0tNCm2+6c+DD2tLAYpZgAuY3YN1uQ E8uXz2u+80F3/Nsc3Y8pkC+qRgSPEt0w= X-Google-Smtp-Source: AGHT+IEASfYYmIHwktibqZ31d9+ZqnKZcMfp8c3E0JMa/ols3Md65NAy2YarqyTYA3pzlqLukpjDCNg6+lxbsijjcg0= X-Received: by 2002:a17:90a:c397:b0:29b:178e:d9cb with SMTP id h23-20020a17090ac39700b0029b178ed9cbmr4981531pjt.44.1709500495651; Sun, 03 Mar 2024 13:14:55 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> <26084.2494.962383.278446@hergotha.csail.mit.edu> In-Reply-To: <26084.2494.962383.278446@hergotha.csail.mit.edu> From: Rick Macklem Date: Sun, 3 Mar 2024 13:14:44 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Garrett Wollman Cc: stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4TnvjY1rCsz4DXs On Sat, Mar 2, 2024 at 9:25=E2=80=AFPM Garrett Wollman wrote: > > < > > I believe this explains why vn_copy_file_range sometimes takes much > > longer than a second: our servers often have lots of data waiting to > > be written to disk, and if the file being copied was recently modified > > (and so is dirty), this might take several seconds. I've set > > vfs.zfs.dmu_offset_next_sync=3D0 on the server that was hurting the mos= t > > and am watching to see if we have more freezes. > > In case anyone is wondering why this is an issue, it's the combination > of two factors: > > 1) vn_generic_copy_file_range() attempts to preserve holes in the > source file. Just fyi, when I was first doing the copy_file_range(2) syscall, the discus= sion seemed to think this was a reasonable thing to do. It is now not so obvious for file systems doing compression, such as ZFS. It happens that ZFS will no longer use vn_generic_copy_file_range() when block cloning is enabled and I have no idea what block cloning does w.r.t. preserving holes. For non-compression file systems, comparing va_size with va_bytes should serve as a reasonable hint w.r.t. the file being sparse. If the file is not sparse, vn_generic_copy_file_range() should not bother doing SEEK_DATA/SEEK_HOLE. (I had intended to do such a patch, but I cannot now remember if I did do s= o. I'll take a look.) Note that this patch would not affect ZFS, but could improve UFS performain= ce where vn_generic_copy_file_range() is used to do the copying. rick > > 2) ZFS does automatic hole-punching on write for filesystems where > compression is enabled. It happens in the same code path as > compression, checksum generation, and redundant-write suppression, and > thus does not happen until the dirty blocks are about to be committed > to disk. So if the file is dirty, ZFS doesn't "know" whether thare > where the then-extant holes are until a sync has completed. > > While vn_generic_copy_file_range() has a flag to stop and return > partial success after a second of copying, this flag does not affect > sleeps internal to the filesystem, so zfs_holey() can sleep > indefinitely and vn_generic_copy_file_range() can't do anything about > it until the sync has already happened. > > -GAWollman > From nobody Sun Mar 3 21:17:30 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tnvmm1lYdz5CLBH for ; Sun, 3 Mar 2024 21:17:44 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tnvml73Vmz4Fm4 for ; Sun, 3 Mar 2024 21:17:43 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102d.google.com with SMTP id 98e67ed59e1d1-299c11b250fso2465585a91.2 for ; Sun, 03 Mar 2024 13:17:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709500662; x=1710105462; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=B7wOW1XtrxWywIVZf0AqKmTjEIgZSWT21nX16aeL+fk=; b=Z6u8MSjsY2vVPH70UgE8CKInXfNhTSpr3hNEEyBS6t43FiHOddjOdZ7Z06j4EIP7TA mbWBT0+CzoLfv5jUnCB/i2QMSmNXZBncNDAGGWLEiUw4TyNzYYPy/v4kY6Ej6oTZwhUh 0ZeieAPmfHSy75Jsev17BBPUQH3gHKsauyH6FMOdKXRnzguH46q7qSqf/cSKdthltoMI sMFSOK8aNC53hwCNIaoeiH03sAPdOt0cLI4mvPoKzR+2kvR+MOdZwcqcSKIHi4DJetO4 Ur7JKHINOylfUaQwYGqUF99M3tnkOnU/m0Qx55lPahsLSQ7jraQ9exBsnBf5sEs1BTRg IFnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709500662; x=1710105462; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B7wOW1XtrxWywIVZf0AqKmTjEIgZSWT21nX16aeL+fk=; b=HXZwoBvTiguba/d5cPJW6CaCs7dk3oxH/U6WO0x2U5nuSIkaeourMIAhqUC/TWLqea UNECE/NKKE40zLml7FgmchXeZTi7Ul4FtTV9FlflXJNu5ZArGZuJWsCaFFsYY06uc1qe bUwK77WEkMw/+yohfzJuvpDGZHuQcRW4XJYOHa0NgqNGt8PlHkHdULGWlf+ONC3DZ9hJ ajFqw5Y+nFe2pJmEG1jj55OBddDwmSHq05qkLjehYgMKgInJIT8LyPAMGhmqem6uVvsq ph7/vf7KsCWRnU8DwCbz/PYlMvpyrCKwHHIheFtATllnL+staTucYU+w5lDFqnYp28Oe Bh7Q== X-Gm-Message-State: AOJu0YzvYGnEnEsG8YclzMGWOsRyUeyQjxUziEF1AsQSiTVhsioX6qPz jCC1GLDh/1HT/tTPzRAIAOZLZuLMGrP2aPJSLOleniCYAkEdscjm0dZqDsS6X919dzRERnBLJH3 qeaUpX55kHk2AE7lRzo96pxSq1w== X-Google-Smtp-Source: AGHT+IFMFZFInxnpWxufkZDdKnuivwmgLfeAqN8ylq3RvIX3Vu9EvshAiiLiJQC/RNM4fr1BqaflJT89t6MdA/L+57E= X-Received: by 2002:a17:90a:2ec8:b0:29b:2eab:6bda with SMTP id h8-20020a17090a2ec800b0029b2eab6bdamr3967342pjs.35.1709500662124; Sun, 03 Mar 2024 13:17:42 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> In-Reply-To: <26083.64612.717082.366639@hergotha.csail.mit.edu> From: Rick Macklem Date: Sun, 3 Mar 2024 13:17:30 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Garrett Wollman Cc: stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Tnvml73Vmz4Fm4 On Sat, Mar 2, 2024 at 8:28=E2=80=AFPM Garrett Wollman wrote: > > > I wrote previously: > > PID TID COMM TDNAME KSTACK > > 997 108481 nfsd nfsd: master mi_switch sleepq_tim= edwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc_r= un nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > > 997 960918 nfsd nfsd: service mi_switch sleepq_tim= edwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc nfs= svc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > 997 962232 nfsd nfsd: service mi_switch _cv_wait t= xg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd_i= octl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANGE = vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc_r= un_internal svc_thread_start fork_exit fork_trampoline > > I spent some time this evening looking at this last stack trace, and > stumbled across the following comment in > sys/contrib/openzfs/module/zfs/dmu.c: > > | /* > | * Enable/disable forcing txg sync when dirty checking for holes with l= seek(). > | * By default this is enabled to ensure accurate hole reporting, it can= result > | * in a significant performance penalty for lseek(SEEK_HOLE) heavy work= loads. > | * Disabling this option will result in holes never being reported in d= irty > | * files which is always safe. > | */ > | int zfs_dmu_offset_next_sync =3D 1; > > I believe this explains why vn_copy_file_range sometimes takes much > longer than a second: our servers often have lots of data waiting to > be written to disk, and if the file being copied was recently modified > (and so is dirty), this might take several seconds. I've set > vfs.zfs.dmu_offset_next_sync=3D0 on the server that was hurting the most > and am watching to see if we have more freezes. > > If this does the trick, then I can delay deploying a new kernel until > April, after my upcoming vacation. Interesting. Please let us know how it goes. And enjoy your vacation, rick > > -GAWollman > From nobody Sun Mar 3 23:27:04 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TnyfH5Sbjz5CX7j for ; Sun, 3 Mar 2024 23:27:19 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TnyfG2RXxz4XXZ for ; Sun, 3 Mar 2024 23:27:18 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=cK1Aqysc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-299d3b09342so2851861a91.2 for ; Sun, 03 Mar 2024 15:27:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709508436; x=1710113236; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=1oaQGn0x8ZN+ZJjd3tQb0ZEt3Mt08cZKGTGF7NOHDHQ=; b=cK1AqyschW7j0DMv9jWxAnfm3E13RWzUKn+KCGyqY6bbuftdPKmQA4WLLxFRcATLDl tzOpwldgiYEFvEqxy5VSF61793Mwb9nT8AW0uRS3LNzLf9YifvaEMFDZyzRcNhEbFcD+ +GSJtz7CPj8hxd7QRySTPEwLDmdDgN9MxjNuyQPeRBTNzBteSpHkhJKKCz2LS2UP44vl SPHQukcXcB3FM09kH3l4LMJlBT9pk7MOc4i2Qbt9v93aWUHlPlQ163EPTebfRha5oz4j iFXEXT+mOexVTPwiPDpigI9BGjocHCRpkseJbHG0KtWGvc+c9/JOXmBghhQfPlugyTGK fg3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709508436; x=1710113236; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1oaQGn0x8ZN+ZJjd3tQb0ZEt3Mt08cZKGTGF7NOHDHQ=; b=Mp0tlJyg7OzuOpGUccAvBGBLf1zW20MOvQynQjUmRBztlBzt86nPez7T/WkZt81P5n Oty+FYIvlLKXQtXPuJElPULlFxhLgMWSlGo32nljeUsPnrEWlEGh+fyliD/T9ftOnmQ7 7NaPdr8H7GUKvPiigzjLLRoF6KrKyHdpCgzcqX3e+DUquah4fuEcXFdjbCYEnAhAGHEy Pr5onT2wMKEnHAhWUgEzaY8FEqKH+JnykjrRwEhNomcIRmDEwSqsy76hGmq0ZY6VjZAh Fr03XiLRe7QuqOhfThK/rLjThMlNMZlXUdGthcevwRGCgrrd1V7ptGFa1lulZYR0kDWE Y7ZQ== X-Gm-Message-State: AOJu0YwuTnPCjisYVSACwoHvjK9k8EHteqTryf1MUXA+IiMVMjxyx4So GEK4WZM4p9jzZs1kGPgsWpqF62ljhZJXeQrnZNgC7Cvf7GyXyv+A14ht3nRNzbT+kWOzoq445Ld u9SoEm09D2z2WQi2pms67K+P9gJ3iSec= X-Google-Smtp-Source: AGHT+IFIVNcwDYr8OmwAU7tNY/z/mkzNyqhQa5pVMBF0JHlRHFAFZK+lwJgNqW0RdtfL/kH32TSX6Tqp8DfEvFCQT1o= X-Received: by 2002:a17:90a:8008:b0:29a:bf47:e133 with SMTP id b8-20020a17090a800800b0029abf47e133mr5958705pjn.23.1709508436438; Sun, 03 Mar 2024 15:27:16 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <26078.50375.679881.64018@hergotha.csail.mit.edu> <26083.64612.717082.366639@hergotha.csail.mit.edu> In-Reply-To: From: Rick Macklem Date: Sun, 3 Mar 2024 15:27:04 -0800 Message-ID: Subject: Re: 13-stable NFS server hang To: Garrett Wollman Cc: stable@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102a:from] X-Rspamd-Queue-Id: 4TnyfG2RXxz4XXZ On Sun, Mar 3, 2024 at 1:17=E2=80=AFPM Rick Macklem wrote: > > On Sat, Mar 2, 2024 at 8:28=E2=80=AFPM Garrett Wollman wrote: > > > > > > I wrote previously: > > > PID TID COMM TDNAME KSTACK > > > 997 108481 nfsd nfsd: master mi_switch sleepq_t= imedwait _sleep nfsv4_lock nfsrvd_dorpc nfssvc_program svc_run_internal svc= _run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common > > > 997 960918 nfsd nfsd: service mi_switch sleepq_t= imedwait _sleep nfsv4_lock nfsrv_setclient nfsrvd_exchangeid nfsrvd_dorpc n= fssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline > > > 997 962232 nfsd nfsd: service mi_switch _cv_wait= txg_wait_synced_impl txg_wait_synced dmu_offset_next zfs_holey zfs_freebsd= _ioctl vn_generic_copy_file_range vop_stdcopy_file_range VOP_COPY_FILE_RANG= E vn_copy_file_range nfsrvd_copy_file_range nfsrvd_dorpc nfssvc_program svc= _run_internal svc_thread_start fork_exit fork_trampoline > > > > I spent some time this evening looking at this last stack trace, and > > stumbled across the following comment in > > sys/contrib/openzfs/module/zfs/dmu.c: > > > > | /* > > | * Enable/disable forcing txg sync when dirty checking for holes with= lseek(). > > | * By default this is enabled to ensure accurate hole reporting, it c= an result > > | * in a significant performance penalty for lseek(SEEK_HOLE) heavy wo= rkloads. > > | * Disabling this option will result in holes never being reported in= dirty > > | * files which is always safe. > > | */ > > | int zfs_dmu_offset_next_sync =3D 1; > > > > I believe this explains why vn_copy_file_range sometimes takes much > > longer than a second: our servers often have lots of data waiting to > > be written to disk, and if the file being copied was recently modified > > (and so is dirty), this might take several seconds. I've set > > vfs.zfs.dmu_offset_next_sync=3D0 on the server that was hurting the mos= t > > and am watching to see if we have more freezes. > > > > If this does the trick, then I can delay deploying a new kernel until > > April, after my upcoming vacation. > Interesting. Please let us know how it goes. Btw, I just tried this for my trivial test and it worked very well. A 1Gbyte file was cpied in two Copy RPCs of 1sec and slightly less than 1sec. So, your vacation may be looking better, rick > > And enjoy your vacation, rick > > > > > -GAWollman > >