From nobody Tue Dec 3 08:34:38 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y2YqQ0HCDz5g55n; Tue, 03 Dec 2024 08:34:42 +0000 (UTC) (envelope-from des@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y2YqP6cCcz4T04; Tue, 3 Dec 2024 08:34:41 +0000 (UTC) (envelope-from des@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1733214881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=21/Yal2rddP3CLe6ROT17bEN22JZns9a2WGVG5cjx7k=; b=LI1wbeQliLBJwmssu4JYOmumz13Aj/Obo0CVsAfaXOr6UOJYZR7hG0blNGROKd9lHWGJtY 5gSY6QuxSO+cRUBJdrVGWvygTTgxrR81z9EqZLeIGeWxrCwV04+G+lynYO2ABFK/OFzyOa EAa3KS64CNUyeYPN+WRgXzrZpJAXXJEOYBy1aCn72xMSnluEtsXdXmfBvhf/9w9OuOw96z V1Mzg0M3uaOG+XnBB/PWrtLVqd4f5vjtsEg886WpPrp4DjhCZ2c7T+X8QxUOGvUBW4Gmlo OBBqi8RnHc1FEAXPUXhYoH+RZwolaseJBBdrQS7/nmz51rA1J8W6EAM8EbSd4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1733214881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=21/Yal2rddP3CLe6ROT17bEN22JZns9a2WGVG5cjx7k=; b=SyQNnbNsun8X3SQOTIXXpqr8ZscV8bJSqYVpeD2xv5YU5fLxoXyRV5HmDRkURCPbHmBSDO 8LJOOLFJw7dxGgFlzRfwJX2mNUj6ZFPVMrtcz5i1iDZuynXZwKXnXU1TPe3CCQ0SQmR585 QsOxxVv3I0taqvo1qLmm3g0l1E0KrAuc/wuT6RNjrIMxkmVyoeapnnLESd/9zCSz907p52 P9bCZGTwBRHQJtLBvp3eEY6XxXR8X8kzJEDGSo7YAI8UWo0QT6dwWBTHkniVs1i3lCWPvi zuG75yqFnEZLVyhF9P0FHoSypYZakJkRcRqMunTSM8oLvVtOAkn1uozxDjI/9A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1733214881; a=rsa-sha256; cv=none; b=R0Wa/hBRmsdRrGEpsi4360LWyfHLiY39+6lsL6zhb2nShKnaIUK1xJzR+gLgynL+6oIlR+ jZy7pkGv1EjXPbyxzVMTDYT2RVaV2Sfs295DSc1sd/ZByzKfRy9+Qtbq7qxbs/pH57yaWN vnsS5l7ExJketDmBRMKmi2fzn27TPgO5piiFg1YUamZCWkgqXKnM7HREMGLqWGIIFcc0j7 2vbWn94jUBr9Gss//lTXXqbQcw1IE1eK2MiLOvr3L9NcNqN6LNJ5Nw9Ro+EwGr+v62/QVQ UkjksVIzHSuspDH+m9b+UWwyNChNBDywSSXnwatBytJCkn72CxrU+0pSHx3ghQ== Received: from ltc.des.dev (88-177-82-251.subs.proxad.net [88.177.82.251]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: des) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Y2YqP5W1RzG9Z; Tue, 3 Dec 2024 08:34:41 +0000 (UTC) (envelope-from des@freebsd.org) Received: by ltc.des.dev (Postfix, from userid 1001) id 21F50602A4; Tue, 03 Dec 2024 09:34:38 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: "Sean C. Farley" Cc: Mark Millard , FreeBSD Current , FreeBSD Mailing List Subject: Re: port binary dumping core on recent head in poudriere [tmpfs corruptions involving blocks of zeros that should not be all zeros] In-Reply-To: <7f4080f5-a371-9940-480b-df13956c76ab@FreeBSD.org> (Sean C. Farley's message of "Thu, 28 Nov 2024 22:54:32 -0500 (EST)") References: <4D541C32-7DF4-4CAC-B31E-D4DD17977154.ref@yahoo.com> <4D541C32-7DF4-4CAC-B31E-D4DD17977154@yahoo.com> <7f4080f5-a371-9940-480b-df13956c76ab@FreeBSD.org> User-Agent: Gnus/5.13 (Gnus v5.13) Date: Tue, 03 Dec 2024 09:34:38 +0100 Message-ID: <86frn517xd.fsf@ltc.des.dev> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable "Sean C. Farley" writes: > I ran all of the tmpfs*.sh tests from HEAD which all pass except for > tmpfs24.sh. > > $ ./all.sh -o tmpfs24.sh > 20241128 22:33:38 all: tmpfs24.sh > Min hole size is 4096, file size is 524288000. > data #1 @ 0, size=3D4096) > hole #2 @ 4096, size=3D4096 > data #3 @ 8192, size=3D4096) > hole #4 @ 12288, size=3D4096 > data #5 @ 16384, size=3D4096) > hole #6 @ 20480, size=3D524267520 > --- /tmp/tmpfs24.exp 2024-11-28 22:33:40.222199000 -0500 > +++ /tmp/tmpfs24.log 2024-11-28 22:33:40.225048000 -0500 > @@ -5,4 +5,3 @@ > hole #4 @ 12288, size=3D4096 > data #5 @ 16384, size=3D4096) > hole #6 @ 20480, size=3D524267520 > -<> > FAIL tmpfs24.sh exit code 1 This appears to be a bug in the test. The lsholes program is incapable of producing the <> line, and the output it does produce is consistent with what the test should expect to find. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org From nobody Tue Dec 3 19:46:09 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y2rkJ12DFz5gf0S for ; Tue, 03 Dec 2024 19:46:16 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp052.goneo.de (smtp5.goneo.de [IPv6:2001:1640:5::8:30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y2rkG4z8Sz4hMc for ; Tue, 3 Dec 2024 19:46:14 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=K92mIwBi; spf=pass (mx1.freebsd.org: domain of freebsd@walstatt-de.de designates 2001:1640:5::8:30 as permitted sender) smtp.mailfrom=freebsd@walstatt-de.de; dmarc=none Received: from hub1.goneo.de (hub1.goneo.de [IPv6:2001:1640:5::8:52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id BBDAC240E9D for ; Tue, 3 Dec 2024 20:46:12 +0100 (CET) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id 24EF5240593 for ; Tue, 3 Dec 2024 20:46:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1733255171; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=86rjhtH5sgYMHURgC4O4b2c6MTmFJteClt0+p+FZpDI=; b=K92mIwBihb7BXtKCv1mr9Mi3QcaOTGmcp0ZXEh7uU7uqN7BRRm2axaAN43S1SRYWlQpXu3 SiZMZebNDkhxoiSGvsqYE654eo7lzEedS/TBK+7fwXEP2Hxulch6o/6YIzsQYYkxq/uvzi VE63oa/DniohbzJfFci6fTwMNyVLV0olIVf/qlx7W0TGs0SoW5GpA4tNYPuirQGzEUSiAk vr7VyMAsxvw/2fuZoApBz0wW6oasxcXXIyzZnVz0LfZozA4EECmkDuRKdFccaBSqy3/MtS U6xNBz771vzVNz9isN/vN5767WRTN5JSVWArSi3t2yCMcABvl+QsPUeDPI5ZFA== Received: from hermann.dmz.walstatt.dynvpn.de (dynamic-2a01-0c22-34b7-bf02-4867-adfb-757d-dea8.c22.pool.telefonica.de [IPv6:2a01:c22:34b7:bf02:4867:adfb:757d:dea8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id D457F240564 for ; Tue, 3 Dec 2024 20:46:10 +0100 (CET) Date: Tue, 3 Dec 2024 20:46:09 +0100 From: FreeBSD User To: FreeBSD CURRENT Subject: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: <20241203204609.68e04364@hermann.dmz.walstatt.dynvpn.de> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: aa4e5b X-Rspamd-UID: 828e64 X-Spamd-Result: default: False [-3.59 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; R_SPF_ALLOW(-0.20)[+ip6:2001:1640:5::8:0/112]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[2001:1640:5::8:30:from]; RCPT_COUNT_ONE(0.00)[1]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:25394, ipnet:2001:1640::/32, country:DE]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; SUBJECT_HAS_EXCLAIM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[walstatt-de.de]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[walstatt-de.de:+] X-Rspamd-Queue-Id: 4Y2rkG4z8Sz4hMc X-Spamd-Bar: --- On most recent CURRENT (on some boxes of ours, not all) fetch/git seem to be stuck forever fetching tarballs from ports, fetching Emails via claws-mail (TLS), opening websites via librewolf and firefox or pulling repositories via git. CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 23:11:07 CET 2024 amd64 When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: error: RPC failed: curl 56 recv failure: Operation timed out Ab initio cloning isn't possible, I had to rsync, but after that, for some hosts pulling ports or /usr/src worked again. Conneting to my Email provider via claws-mail results in logon failures, opening websites via librewolf/firefox (youtube, i.e.) results in a stuck connection - I have to open the site repeatedly to open it by chance. I'm out of ideas and this drives me nuts! In the lab I updated today several other hosts to most recent CURRENT and those machines do not show this strange behaviour, but they open and pull sources via "git -C /usr/src pull" somehow slow (it take a while until the connection opens). The only thing in common of those boxes in question is an older Intel NIC, em(8) based. I can not pinpoint the problem, it occured 1st of December. Sorry for the fuzzy description of the problem, I might be of use of some hints how to investigate more deeper Kind regards and thanks in advance Oliver From nobody Wed Dec 4 12:38:02 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y3H9z2V6Gz5fVxg for ; Wed, 04 Dec 2024 12:38:15 +0000 (UTC) (envelope-from ax61@disroot.org) Received: from layka.disroot.org (layka.disroot.org [178.21.23.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y3H9x1mSbz4Cck for ; Wed, 4 Dec 2024 12:38:13 +0000 (UTC) (envelope-from ax61@disroot.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=disroot.org header.s=mail header.b=HdGuEZOI; spf=pass (mx1.freebsd.org: domain of ax61@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=ax61@disroot.org; dmarc=pass (policy=reject) header.from=disroot.org Received: from mail01.disroot.lan (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 768CE24D0D for ; Wed, 4 Dec 2024 13:38:06 +0100 (CET) X-Virus-Scanned: SPAM Filter at disroot.org Received: from layka.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id ta_a1hgRskOe for ; Wed, 4 Dec 2024 13:38:02 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1733315882; bh=1REdhYbBAJsYZJD/ZM+CxCKiIEXEcvZpf8Hb/Bvrpnw=; h=Date:From:To:Subject; b=HdGuEZOIYH47UCZnzPbMkuZjGPaLLe1VaYOaBspyRPeIaQ5OW1g1+ldmDvyy+zfbd JOMa3OvEjzwknwgYfU5HsfCHv5uuCsViEgUCdNTNVF4CHiiW/ADD0iqBkwscDflsdM mMDGHteQl4tTkz0Zlu1XIshE7hR7WDG6zheFW+SQ8JkwBM2b+XlealguM62FZBHzDt vupKGT+aqIakJWYH6wD2J+S4QwshMu/F4iujQ99tAbU5AaAbWGQeP6xq8EWdfR6hs1 3B7zYDxWXrYw47qgCjNRHy0ja7h32BPwMQW8A2+d3TXnqC75tMd8+KeH89KtTGdnot cJaaP2D/lkxgA== List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Date: Wed, 04 Dec 2024 12:38:02 +0000 From: ax61@disroot.org To: freebsd-current@freebsd.org Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: X-Sender: ax61@disroot.org Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [1.58 / 15.00]; RBL_VIRUSFREE_BOTNET(2.00)[178.21.23.139:from]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.998]; NEURAL_SPAM_LONG(0.57)[0.572]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; DMARC_POLICY_ALLOW(0.00)[disroot.org,reject]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; R_DKIM_ALLOW(0.00)[disroot.org:s=mail]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; SUBJECT_HAS_EXCLAIM(0.00)[]; DKIM_TRACE(0.00)[disroot.org:+]; MISSING_XM_UA(0.00)[]; FROM_NO_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; R_SPF_ALLOW(0.00)[+a]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4Y3H9x1mSbz4Cck X-Spamd-Bar: + In short, I do not know the reason; only noting that had seen earlier. Recreating from https://lists.freebsd.org/archives/freebsd-current/2024-December/006778.html ... On 20241203, FreeBSD User wrote ... > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > 23:11:07 CET 2024 > amd64 > > When performing "git pull" und /usr/ports, I received after roughly 5-7 > minutes: > > error: RPC failed: curl 56 recv failure: Operation timed out > > Ab initio cloning isn't possible On 20240722 and some days earlier, in USA, I had observed (similarly for "src") on "stable/14" installation ... # git clone --bare -v -v https://git.FreeBSD.org/ports.git ports.git Cloning into bare repository 'ports.git'... POST git-upload-pack (175 bytes) want 2183117a718ecc35b10e9d4af067dc939e9d1f41 (HEAD) want 5f4d6e1d6b07bb34e5cbf866b4ef81e8ccdcb2da (refs/heads/2014Q1) ... want de75d91a717bb5aa8399afe985c79460d259d692 (refs/tags/release/9.0.0) want 6e96063d1630031e4929aba1f5da06983b72aa00 (refs/tags/release/9.1.0) want 2c356644f01f5fb7135e445863e91d90010dd022 (refs/tags/release/9.2.0) want ae51d1f5a3d9224c137eb66b72454b43a33daae3 (refs/tags/release/9.3.0) POST git-upload-pack (gzip 7202 to 3624 bytes) remote: Enumerating objects: 6274138, done. remote: Counting objects: 100% (28094/28094), done. remote: Compressing objects: 100% (21225/21225), done. error: RPC failed; curl 18 transfer closed with outstanding read data remaining error: 524 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output > , I had to rsync, but after that, for some hosts pulling > ports or /usr/src worked again. Hunh, interesting. I did not have any other tree to "rsync" from, nor would have the thought occurred to me. After few days the issue was gone by itself (or, have not seen since). > Conneting to my Email provider via claws-mail results in logon > failures, opening websites > via librewolf/firefox (youtube, i.e.) results in a stuck connection - I > have to open the > site repeatedly to open it by chance. Ok, I did not have any issue connecting to elsewhere. -- From nobody Wed Dec 4 17:20:39 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y3PSH31TFz5fv94 for ; Wed, 04 Dec 2024 17:21:03 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from fout-a8-smtp.messagingengine.com (fout-a8-smtp.messagingengine.com [103.168.172.151]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y3PSG571lz4p0h for ; Wed, 4 Dec 2024 17:21:02 +0000 (UTC) (envelope-from dch@skunkwerks.at) Authentication-Results: mx1.freebsd.org; none Received: from phl-compute-02.internal (phl-compute-02.phl.internal [10.202.2.42]) by mailfout.phl.internal (Postfix) with ESMTP id D2B7613806FC; Wed, 4 Dec 2024 12:21:01 -0500 (EST) Received: from phl-imap-02 ([10.202.2.81]) by phl-compute-02.internal (MEProxy); Wed, 04 Dec 2024 12:21:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1733332861; x=1733419261; bh=M19TfVtMsIWDvupMq2hb5wsBSZ/arqNNQKwCPt6fyLk=; b= SryuAbwjcUV4s96231RuSznS8UvsFzPF/5uXkMgxcWmx757fZWKadHw0KFv+9jFG xAePKpZZ2SUL1KT4m1eICGGIJlfPw+fWIdDwbPBHmNR9KSL+Z2uSl4/jRgUlE1tt 2KU8RY+wyD2xrtA10hUvnRy3xTk5eVbOgiWu7e/cj2FtMhBT+OLB+1EPlx+WQMWH M+67Q/b3/V570UqLv5wVA1r2I8X28LPDZC5GOm9+vgyc34MHk5lzKNPjd3oda1a0 MwcBXxw3pB/1iCVhYf52zIVqdJeOO8+eNCP2PgXKyABi3zv3egQwPCxKM0WFVUR7 PJDqxcxsDH9+jG72LbeKAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1733332861; x=1733419261; bh=M 19TfVtMsIWDvupMq2hb5wsBSZ/arqNNQKwCPt6fyLk=; b=GVGJRSeAEE1OWwiyD 3zMIW0sss/rl/oXt4bsBsyCDNNsdfiMtKX8hX15LLuJZkwnT28AdguZUzTEc6Uqe FJy2f3+rPLDiA3fq/QtosSUz4adDOt4LoOeBrU7AFOGTNOKY33t9n3d0kL9BOtkE 4J2ZTWL9WzyiElGYDAfISOWN1Ws6h6eXIy+5WH5OTPgJ0RO/Ehm8K1+26n8QcRJL J9/AAoyEZbGZy3LWFXTjCohZLtd7LiDdWDirnxwsJmUF+l7CWIxEEr303RHaSAk6 9lCOuywVt00UqHOgaYUoZuhCmBxLqmCHrvNDgWBxkSJ6ySBFkBPyryRvJhxpHBdp E21UA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrieehgdeliecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpefoggffhffvkfgjfhfutgfgsehtjeertdertddtnecu hfhrohhmpedfffgrvhgvucevohhtthhlvghhuhgsvghrfdcuoegutghhsehskhhunhhkfi gvrhhkshdrrghtqeenucggtffrrghtthgvrhhnpefgjeetffdutdduhedttefgteegvdeh feefgfektdefhfekudevhfefheekveeuteenucffohhmrghinheptghurhhlrdguvghvpd hsthgrtghkohhvvghrfhhlohifrdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfr rghrrghmpehmrghilhhfrhhomhepuggthhesshhkuhhnkhifvghrkhhsrdgrthdpnhgspg hrtghpthhtohepvddpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepfhhrvggvsghs ugdqtghurhhrvghnthesfhhrvggvsghsugdrohhrghdprhgtphhtthhopehfrhgvvggssh guseifrghlshhtrghtthdquggvrdguvg X-ME-Proxy: Feedback-ID: ic0e84090:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id 34268B0006A; Wed, 4 Dec 2024 12:21:01 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Date: Wed, 04 Dec 2024 17:20:39 +0000 From: "Dave Cottlehuber" To: "FreeBSD User" , freebsd-current Message-Id: <6626e5c0-ba01-4966-a28c-82a25251ca3f@app.fastmail.com> In-Reply-To: <20241203204609.68e04364@hermann.dmz.walstatt.dynvpn.de> References: <20241203204609.68e04364@hermann.dmz.walstatt.dynvpn.de> Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US] X-Rspamd-Queue-Id: 4Y3PSG571lz4p0h X-Spamd-Bar: ---- On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem > to be stuck > forever fetching tarballs from ports, fetching Emails via claws-mail > (TLS), opening > websites via librewolf and firefox or pulling repositories via git. > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > 23:11:07 CET 2024 > amd64 > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: > > error: RPC failed: curl 56 recv failure: Operation timed out Generally it would be worth seeing if the HTTP(S) layers are doing the right thing or not, and then working down from there, to tcpdump / wireshark and then if necessary into kernel itself. If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball, and try it again in curl, with tcpdump already prepared to capture traffic to the remote host. tcpdump -w /tmp/curl.pcap -i ... host ... env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace /tmp/curl.log https://what.ev/er I would guess that between the two something useful should pop up. I like opening the pcap in wireshark, it often has angry red and black highlighted lines already giving me a hint. The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS traffic as well in case there are issues further in. Very handy, see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. If your issues only occur with git pull, its also curl inside and supports similar debugging. Ferreting through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. A+ Dave From nobody Thu Dec 5 10:51:03 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y3rlz6XrDz5gJbB for ; Thu, 05 Dec 2024 10:51:11 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp052.goneo.de (smtp5.goneo.de [IPv6:2001:1640:5::8:30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y3rlz4Wp5z4kDJ for ; Thu, 5 Dec 2024 10:51:11 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Authentication-Results: mx1.freebsd.org; none Received: from hub1.goneo.de (hub1.goneo.de [85.220.129.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id 072D8240E51; Thu, 5 Dec 2024 11:51:08 +0100 (CET) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id 2DB182402D8; Thu, 5 Dec 2024 11:51:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1733395866; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w9mqz/VRZUfR7JG2pXywXz0fqchkkz73/ZV+e0YKOrA=; b=fmvnvL30/+uzhJXE6p17Zer/9qQw1dpJogt2wUeMK9pni1TTFGQUmTdNb9Y7cKSYNLoT0k L4IoWclVAzwN5Ww9doXlr8xYk7DAhbwtaWx2PtIOaVoSDJM/JEHQI3DtIHn7jnkKvEPeUA QtT0x1PvWvDrRwOIG6oqtXhIQel8IwsCwUhDobOwX6EdqyhpAg3gGTMQGOhw0AFk/mFb9N 9JgtsGSZn57PSn+WhjNeF5h8nw6bXgHZDuHiy6uy3WHxkxFBCSYN7whTrJpcvjga0lBTSY zqznRVrAqtEQtSAaER2MAL5jbTsYXIIHDSjcVHNzOH2oy7NYLK9QrMsvMBquxw== Received: from hermann.intern.walstatt.dynvpn.de (dynamic-077-183-116-031.77.183.pool.telefonica.de [77.183.116.31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id DBE472402D6; Thu, 5 Dec 2024 11:51:05 +0100 (CET) Date: Thu, 5 Dec 2024 11:51:03 +0100 From: FreeBSD User To: "Dave Cottlehuber" Cc: freebsd-current Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: <20241205115103.1ed6d7f6@hermann.intern.walstatt.dynvpn.de> In-Reply-To: <6626e5c0-ba01-4966-a28c-82a25251ca3f@app.fastmail.com> References: <20241203204609.68e04364@hermann.dmz.walstatt.dynvpn.de> <6626e5c0-ba01-4966-a28c-82a25251ca3f@app.fastmail.com> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: ada1e8 X-Rspamd-UID: b1c219 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:25394, ipnet:2001:1640::/32, country:DE] X-Rspamd-Queue-Id: 4Y3rlz4Wp5z4kDJ X-Spamd-Bar: ---- On Wed, 04 Dec 2024 17:20:39 +0000 "Dave Cottlehuber" wrote: Thank you very much for responding! > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem > > to be stuck > > forever fetching tarballs from ports, fetching Emails via claws-mail > > (TLS), opening > > websites via librewolf and firefox or pulling repositories via git. > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > > 23:11:07 CET 2024 > > amd64 > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: > > > > error: RPC failed: curl 56 recv failure: Operation timed out > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing or > not, and then working down from there, to tcpdump / wireshark and then if necessary > into kernel itself. My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and theory,of course). I tried due to "a feeling" my used older Intel based NIC could have some checksum issues like in the past (I saw e1000 driver updates recently flowing into FreeBSD CURRENT). > > If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball, > and try it again in curl, with tcpdump already prepared to capture traffic to the > remote host. > > tcpdump -w /tmp/curl.pcap -i ... host ... > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace > /tmp/curl.log https://what.ev/er > > I would guess that between the two something useful should pop up. > > I like opening the pcap in wireshark, it often has angry red and black highlighted > lines already giving me a hint. > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS traffic > as well in case there are issues further in. Very handy, > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. > > If your issues only occur with git pull, its also curl inside and supports similar > debugging. Ferreting > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. > > A+ > Dave > Thanks for the hints and precious tips! I'll digg deeper into the matter. In the meanwhile, I updated some other machines running CURRENT since approx. two weeks with an older CURRENT to the most recent one - and face similar but not identical problems! Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems except they take longer to accomplish than expected. Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout. On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), cloning attempts for 14.2-RELENG ends up in this mess: # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/ Cloning into '14.2-RELENG/src'... error: RPC failed; curl 56 Recv failure: Operation timed out fatal: expected 'packfile' This is nasty. The host now in question has an i350 based dual-port NIC - the host's kernel is very similar to the box I reported the issue first time, both do have customized kernels (in most cases, I compile several modules like ZFS and several NETGRAPH modules statically into the kernel - a habit inherited from a small FBSD project I configured (I wouldn't say developed) which does not allow loadable kernel modules due to regulations. I hoped others would stumble over this tripwire in recent CURRENT sources, since the phenomena and its distribution over a bunch of CURRENT boxes with different OS states seemingly show different behviour. And for the record: I also build my ports via poudriere and mostly via make. I also rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and so on to ensure having latest sources/packages. (I repeat myself here again, sorry, its for the record). Will report in on further development and "investigations" Kind regards and thanks, oh From nobody Thu Dec 5 15:55:00 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y3zVf0VY2z5ggH9 for ; Thu, 05 Dec 2024 15:55:06 +0000 (UTC) (envelope-from tamelingdaniel@gmail.com) Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y3zVd1rmTz3xB2 for ; Thu, 5 Dec 2024 15:55:05 +0000 (UTC) (envelope-from tamelingdaniel@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=algA1LAn; spf=pass (mx1.freebsd.org: domain of tamelingdaniel@gmail.com designates 2a00:1450:4864:20::62f as permitted sender) smtp.mailfrom=tamelingdaniel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x62f.google.com with SMTP id a640c23a62f3a-aa1e6ecd353so165326466b.1 for ; Thu, 05 Dec 2024 07:55:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733414102; x=1734018902; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=kjyfxYNNZDc+NH+SmiTaSl5+hGAaKXTYSiokRshpguY=; b=algA1LAnsXUsDNQa6bGC3SEjQvvx+YBNtsTtb1ao3VnbolXm/IggzYNZZb9httcQEl h2O6zLCPRAVk5LXGyKR8TDkgOhyH5Dq2+9rtr4sPz3ZmToZdX26FVN+DmQXnSXvqXp2a IDaXtPTC2dMUUZE+8dd/KGsns5bUbTkcrE/dorqiHMu6ivQj+1wr5ECM7J7D3aeTzPTd 7N2qPTQdpf99cIkOmUctJcvDiFMqx0xeFrU4rFXuzaSK3EsfV5uoNLiErnG3GEsTNdsz 94lw9aJ9Uo1zdTxrvIY2yC8LgY4Jmnovq/Cxw1pslLGO9UidurCnFt/DJpXJDQst12Y4 aykQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733414102; x=1734018902; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=kjyfxYNNZDc+NH+SmiTaSl5+hGAaKXTYSiokRshpguY=; b=VVTnMentTD3C54gpHTp0twChgDmYUMK42icYnj2TEz5R4/FUYiPUb6AzhReoJmqIHn nzis4I1YImfmEQ8q5HyMtKOAovX2CZ/Yz9dnBfP0VnnLlkWQGsBw9DubNB3KyaOuQZXg eRcImwauCT58l0eHjy2toDMDdKuCM7s6YvjjoIi6Gk6vRB1qdo3ONL8Zb7tuwJHHXdFH Wyr8bYUVJjA1ZJf7wPSr6ZESsGK84q26WJr8or9bLBQ/L4De9vFj5FaS30q7Q2VmSAwh shP/Fh3oQvzLqJOjIitCLB6ZqHmiHKE88csLwBaq2NU+55hm9mLnLw4DPJ6GHoVNFIpZ RqwA== X-Gm-Message-State: AOJu0YzHhD3QH6BxfQ4fvAPBzoLSv2d/kvSXDkZDYnHYJ4yem/Ouj48c xZQ8L0ZsRByB/n1R9GVpEdo6YNU0kWqqC8j+YyVeHWRvGtl+5AYq1RqSUA== X-Gm-Gg: ASbGncvDu88MMWfy3goJp5gKxHE8l5/HdDdvyjX9oxpzDRlQ5fiv7+m28LmbZlpzMN8 i41f3Lc+m1+OziOPoAYBEscEdPXki1GR0CDMavWdFRFHwWFii5UQ3u7+P5bOUkETi8Ef0YSokst uD2+YDaCu3pGpP5wAHFuCT/zO/2YCQlU9PHlm7K4JRsz158fq7fc8loqq/oXbuZ9pJxzwYevo5A 0sxVB1fSqnACbRN90SXjHQtVlwIl9yXwIovrfJTEhKWQqUgi1ParLdd X-Google-Smtp-Source: AGHT+IH/WhYrOdn4vZoecf7iX7JszQ0xDM7C9IGNMW0FPKKhPe2X4DE1mLG2nminTthzvtNaWvRTVg== X-Received: by 2002:a05:6402:34d5:b0:5d0:e9b7:170c with SMTP id 4fb4d7f45d1cf-5d10cb5c39fmr15791488a12.18.1733414102194; Thu, 05 Dec 2024 07:55:02 -0800 (PST) Received: from localhost ([2001:9e8:c402:7000:5ec9:c431:7277:2312]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa6260ae4f3sm108460366b.159.2024.12.05.07.55.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Dec 2024 07:55:01 -0800 (PST) Date: Thu, 5 Dec 2024 16:55:00 +0100 From: Daniel Tameling To: freebsd-current@freebsd.org Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: References: <20241203204609.68e04364@hermann.dmz.walstatt.dynvpn.de> <6626e5c0-ba01-4966-a28c-82a25251ca3f@app.fastmail.com> <20241205115103.1ed6d7f6@hermann.intern.walstatt.dynvpn.de> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20241205115103.1ed6d7f6@hermann.intern.walstatt.dynvpn.de> X-Spamd-Result: default: False [-3.96 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.959]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; SUBJECT_HAS_EXCLAIM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62f:from] X-Rspamd-Queue-Id: 4Y3zVd1rmTz3xB2 X-Spamd-Bar: --- On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote: > On Wed, 04 Dec 2024 17:20:39 +0000 > "Dave Cottlehuber" wrote: > > Thank you very much for responding! > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem > > > to be stuck > > > forever fetching tarballs from ports, fetching Emails via claws-mail > > > (TLS), opening > > > websites via librewolf and firefox or pulling repositories via git. > > > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > > > 23:11:07 CET 2024 > > > amd64 > > > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: > > > > > > error: RPC failed: curl 56 recv failure: Operation timed out > > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing or > > not, and then working down from there, to tcpdump / wireshark and then if necessary > > into kernel itself. > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and > theory,of course). I tried due to "a feeling" my used older Intel based NIC could have > some checksum issues like in the past (I saw e1000 driver updates recently flowing into > FreeBSD CURRENT). > > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball, > > and try it again in curl, with tcpdump already prepared to capture traffic to the > > remote host. > > > > tcpdump -w /tmp/curl.pcap -i ... host ... > > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace > > /tmp/curl.log https://what.ev/er > > > > I would guess that between the two something useful should pop up. > > > > I like opening the pcap in wireshark, it often has angry red and black highlighted > > lines already giving me a hint. > > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS traffic > > as well in case there are issues further in. Very handy, > > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. > > > > If your issues only occur with git pull, its also curl inside and supports similar > > debugging. Ferreting > > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. > > > > A+ > > Dave > > > > Thanks for the hints and precious tips! I'll digg deeper into the matter. > > In the meanwhile, I updated some other machines running CURRENT since approx. two weeks > with an older CURRENT to the most recent one - and face similar but not identical > problems! > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems > except they take longer to accomplish than expected. > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout. > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), cloning > attempts for 14.2-RELENG ends up in this mess: > > # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/ > Cloning into '14.2-RELENG/src'... > error: RPC failed; curl 56 Recv failure: Operation timed out > fatal: expected 'packfile' > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's > kernel is very similar to the box I reported the issue first time, both do have > customized kernels (in most cases, I compile several modules like ZFS and > several NETGRAPH modules statically into the kernel - a habit inherited from a small FBSD > project I configured (I wouldn't say developed) which does not allow loadable kernel > modules due to regulations. > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the > phenomena and its distribution over a bunch of CURRENT boxes with different OS states > seemingly show different behviour. > > And for the record: I also build my ports via poudriere and mostly via make. I also > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and so > on to ensure having latest sources/packages. > > (I repeat myself here again, sorry, its for the record). > > Will report in on further development and "investigations" > > Kind regards and thanks, > > oh > > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some networking issues that got fixed later. Otherwise I would start with ping and traceroute to figure out if they show this issue and where it occurs. -- Best regards, Daniel From nobody Thu Dec 5 16:33:54 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y40Mj0Qjyz5dlM2 for ; Thu, 05 Dec 2024 16:34:09 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp6.goneo.de (smtp6.goneo.de [IPv6:2001:1640:5::8:31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y40Mh52P5z436Q for ; Thu, 5 Dec 2024 16:34:08 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Authentication-Results: mx1.freebsd.org; none Received: from hub1.goneo.de (hub1.goneo.de [85.220.129.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp6.goneo.de (Postfix) with ESMTPS id 0405C240D59; Thu, 5 Dec 2024 17:33:59 +0100 (CET) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id 234412402D6; Thu, 5 Dec 2024 17:33:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1733416437; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Iax3Dt/7wqw4yqL3X4GVNj9S2MuoIl6fkTnhJ7IUJ+8=; b=gdtq60DKWDkW9pmgmi662N0uiKu+K1XkrVnpiosGPDVmlz9+UqTfnUq3nTvFWNjnyr6L9M ChbthmvFsGPT1jXxGBT4i6A/wwL1SswiCff8ToUOGfO00Tjer6mcKgLNYYbQx/m/MHvP8O GY/3wXwm17e/PDWqg56zqwdhZkHa3BREu5y+6XPXhFzj2mxogbuDIm9Hqk6kvk/5KpOEEB A/hZKlWEDN84FhBf6YUyE6+SdcziYMzOA5Il530N853T9O5Tw1yXQWVOqSY/iMGrPEzbck Tu8q1QFwzrP28u9djwZXpRAdFI+gpSvdygbT+Si68/MnZHvLO99lR389Vmx39g== Received: from hermann.intern.walstatt.dynvpn.de (dynamic-077-183-116-031.77.183.pool.telefonica.de [77.183.116.31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id CB74E24029B; Thu, 5 Dec 2024 17:33:56 +0100 (CET) Date: Thu, 5 Dec 2024 17:33:54 +0100 From: FreeBSD User To: Daniel Tameling Cc: freebsd-current@freebsd.org Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: <20241205173354.23c4e592@hermann.intern.walstatt.dynvpn.de> In-Reply-To: References: <20241203204609.68e04364@hermann.dmz.walstatt.dynvpn.de> <6626e5c0-ba01-4966-a28c-82a25251ca3f@app.fastmail.com> <20241205115103.1ed6d7f6@hermann.intern.walstatt.dynvpn.de> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: 82e4cd X-Rspamd-UID: 2ceb5b X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:25394, ipnet:2001:1640::/32, country:DE] X-Rspamd-Queue-Id: 4Y40Mh52P5z436Q X-Spamd-Bar: ---- On Thu, 5 Dec 2024 16:55:00 +0100 Daniel Tameling wrote: > On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote: > > On Wed, 04 Dec 2024 17:20:39 +0000 > > "Dave Cottlehuber" wrote: > > > > Thank you very much for responding! > > > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: > > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem > > > > to be stuck > > > > forever fetching tarballs from ports, fetching Emails via claws-mail > > > > (TLS), opening > > > > websites via librewolf and firefox or pulling repositories via git. > > > > > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > > > > 23:11:07 CET 2024 > > > > amd64 > > > > > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: > > > > > > > > error: RPC failed: curl 56 recv failure: Operation timed out > > > > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing > > > or not, and then working down from there, to tcpdump / wireshark and then if > > > necessary into kernel itself. > > > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and > > theory,of course). I tried due to "a feeling" my used older Intel based NIC could have > > some checksum issues like in the past (I saw e1000 driver updates recently flowing > > into FreeBSD CURRENT). > > > > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball, > > > and try it again in curl, with tcpdump already prepared to capture traffic to the > > > remote host. > > > > > > tcpdump -w /tmp/curl.pcap -i ... host ... > > > > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace > > > /tmp/curl.log https://what.ev/er > > > > > > I would guess that between the two something useful should pop up. > > > > > > I like opening the pcap in wireshark, it often has angry red and black highlighted > > > lines already giving me a hint. > > > > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS > > > traffic as well in case there are issues further in. Very handy, > > > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. > > > > > > If your issues only occur with git pull, its also curl inside and supports similar > > > debugging. Ferreting > > > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. > > > > > > A+ > > > Dave > > > > > > > Thanks for the hints and precious tips! I'll digg deeper into the matter. > > > > In the meanwhile, I updated some other machines running CURRENT since approx. two > > weeks with an older CURRENT to the most recent one - and face similar but not > > identical problems! > > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems > > except they take longer to accomplish than expected. > > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout. > > > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD > > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), cloning > > attempts for 14.2-RELENG ends up in this mess: > > > > # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/ > > Cloning into '14.2-RELENG/src'... > > error: RPC failed; curl 56 Recv failure: Operation timed out > > fatal: expected 'packfile' > > > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's > > kernel is very similar to the box I reported the issue first time, both do have > > customized kernels (in most cases, I compile several modules like ZFS and > > several NETGRAPH modules statically into the kernel - a habit inherited from a small > > FBSD project I configured (I wouldn't say developed) which does not allow loadable > > kernel modules due to regulations. > > > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the > > phenomena and its distribution over a bunch of CURRENT boxes with different OS states > > seemingly show different behviour. > > > > And for the record: I also build my ports via poudriere and mostly via make. I also > > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and > > so on to ensure having latest sources/packages. > > > > (I repeat myself here again, sorry, its for the record). > > > > Will report in on further development and "investigations" > > > > Kind regards and thanks, > > > > oh > > > > > > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some > networking issues that got fixed later. No, pure Hardware and FreeBSD ... > > Otherwise I would start with ping and traceroute to figure out if they show this issue > and where it occurs. > From nobody Fri Dec 6 02:46:42 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y4FzB2zqvz5g7jl; Fri, 06 Dec 2024 02:47:18 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp052.goneo.de (smtp5.goneo.de [IPv6:2001:1640:5::8:30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y4Fz82G6bz46Wx; Fri, 6 Dec 2024 02:47:16 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=mgJS8hCF; spf=pass (mx1.freebsd.org: domain of freebsd@walstatt-de.de designates 2001:1640:5::8:30 as permitted sender) smtp.mailfrom=freebsd@walstatt-de.de; dmarc=none Received: from hub1.goneo.de (hub1.goneo.de [IPv6:2001:1640:5::8:52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id 7C7C7240E6F; Fri, 6 Dec 2024 03:47:12 +0100 (CET) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id 9F0822402D6; Fri, 6 Dec 2024 03:47:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1733453230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XdFGH3CMbtptjDEBjebyi+V9iZ7X3JTk/OqwmFdpgmU=; b=mgJS8hCFJs1ZRlboyR+Q++xSfQLhEpvyRnRoSANQv5OWwz/TvNkOO054tFMj8t9BevhgIa WeIN37r3sw/yFZ7TYvMwie+0NGrXVViMNVFSaty/5M230CjUREQg3uUHAecOqs/3NnxlEW exa5ZVkazLJShybzuTnxiYhgE/QbOctY1US+ZvS7j1/ISKFQ6YH73XQwiq3gBW4vQZlfoe JcJxx+nVT5c9jZZziwMxb4wrrNRqn0I/4+7qDSwRayAcqIdsSwKuGQ8EnlMDLXTfW77Srh PcHFPr9KTiJTlu7Ne39chbzjmgG99KyuH5bFRVBAPyR8G59e14VKqkt6HqM6zw== Received: from thor.intern.walstatt.dynvpn.de (dynamic-078-054-013-094.78.54.pool.telefonica.de [78.54.13.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id 56A6024029B; Fri, 6 Dec 2024 03:47:10 +0100 (CET) Date: Fri, 6 Dec 2024 03:46:42 +0100 From: FreeBSD User To: freebsd-current@freebsd.org, freebsd-ipfw@freebsd.org Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: <20241206034709.4dd32cc5@thor.intern.walstatt.dynvpn.de> In-Reply-To: <20241205173354.23c4e592@hermann.intern.walstatt.dynvpn.de> References: <20241203204609.68e04364@hermann.dmz.walstatt.dynvpn.de> <6626e5c0-ba01-4966-a28c-82a25251ca3f@app.fastmail.com> <20241205115103.1ed6d7f6@hermann.intern.walstatt.dynvpn.de> <20241205173354.23c4e592@hermann.intern.walstatt.dynvpn.de> Organization: walstatt-de.de List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: a8d101 X-Rspamd-UID: 61e20b X-Spamd-Result: default: False [-3.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; R_SPF_ALLOW(-0.20)[+ip6:2001:1640:5::8:0/112]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[2001:1640:5::8:30:from]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ORG_HEADER(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:25394, ipnet:2001:1640::/32, country:DE]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org,freebsd-ipfw@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[walstatt-de.de]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; DKIM_TRACE(0.00)[walstatt-de.de:+] X-Rspamd-Queue-Id: 4Y4Fz82G6bz46Wx X-Spamd-Bar: --- Am Thu, 5 Dec 2024 17:33:54 +0100 FreeBSD User schrieb: I found the culprit! Disabling IPFW ("ipfw disable firewall") turns system back to normal! For the record: on recent CURRENT, since approx. Nov. 30 and/or December 1st CURRENT seems to corrupt network connections. IPFW is compiled statically into the kernel. The problem sketched below can be reproduced in a more or less obvious manner on recent CURRENT: git pull/git clone of a regular FreeBSD source repo or ports via git+https takes either a couple of time (up to several mintes to initiate the pull) - or, in some worse cases here, the box runs into error: RPC failed; curl 56 Recv failure: Operation timed out claws-mail complains about "corrupted/broken stream", fetching emails takes Aeons - forever, the client does not come back even after several hours. > On Thu, 5 Dec 2024 16:55:00 +0100 > Daniel Tameling wrote: > > > On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote: > > > On Wed, 04 Dec 2024 17:20:39 +0000 > > > "Dave Cottlehuber" wrote: > > > > > > Thank you very much for responding! > > > > > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: > > > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem > > > > > to be stuck > > > > > forever fetching tarballs from ports, fetching Emails via claws-mail > > > > > (TLS), opening > > > > > websites via librewolf and firefox or pulling repositories via git. > > > > > > > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > > > > > 23:11:07 CET 2024 > > > > > amd64 > > > > > > > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: > > > > > > > > > > error: RPC failed: curl 56 recv failure: Operation timed out > > > > > > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing > > > > or not, and then working down from there, to tcpdump / wireshark and then if > > > > necessary into kernel itself. > > > > > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and > > > theory,of course). I tried due to "a feeling" my used older Intel based NIC could have > > > some checksum issues like in the past (I saw e1000 driver updates recently flowing > > > into FreeBSD CURRENT). > > > > > > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball, > > > > and try it again in curl, with tcpdump already prepared to capture traffic to the > > > > remote host. > > > > > > > > tcpdump -w /tmp/curl.pcap -i ... host ... > > > > > > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace > > > > /tmp/curl.log https://what.ev/er > > > > > > > > I would guess that between the two something useful should pop up. > > > > > > > > I like opening the pcap in wireshark, it often has angry red and black highlighted > > > > lines already giving me a hint. > > > > > > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS > > > > traffic as well in case there are issues further in. Very handy, > > > > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. > > > > > > > > If your issues only occur with git pull, its also curl inside and supports similar > > > > debugging. Ferreting > > > > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. > > > > > > > > A+ > > > > Dave > > > > > > > > > > Thanks for the hints and precious tips! I'll digg deeper into the matter. > > > > > > In the meanwhile, I updated some other machines running CURRENT since approx. two > > > weeks with an older CURRENT to the most recent one - and face similar but not > > > identical problems! > > > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems > > > except they take longer to accomplish than expected. > > > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout. > > > > > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD > > > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), cloning > > > attempts for 14.2-RELENG ends up in this mess: > > > > > > # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/ > > > Cloning into '14.2-RELENG/src'... > > > error: RPC failed; curl 56 Recv failure: Operation timed out > > > fatal: expected 'packfile' > > > > > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's > > > kernel is very similar to the box I reported the issue first time, both do have > > > customized kernels (in most cases, I compile several modules like ZFS and > > > several NETGRAPH modules statically into the kernel - a habit inherited from a small > > > FBSD project I configured (I wouldn't say developed) which does not allow loadable > > > kernel modules due to regulations. > > > > > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the > > > phenomena and its distribution over a bunch of CURRENT boxes with different OS states > > > seemingly show different behviour. > > > > > > And for the record: I also build my ports via poudriere and mostly via make. I also > > > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and > > > so on to ensure having latest sources/packages. > > > > > > (I repeat myself here again, sorry, its for the record). > > > > > > Will report in on further development and "investigations" > > > > > > Kind regards and thanks, > > > > > > oh > > > > > > > > > > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some > > networking issues that got fixed later. > > No, pure Hardware and FreeBSD ... > > > > > Otherwise I would start with ping and traceroute to figure out if they show this issue > > and where it occurs. > > > > -- O. Hartmann From nobody Fri Dec 6 16:13:53 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y4bt668D2z5gVMN for ; Fri, 06 Dec 2024 16:14:06 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y4bt55XgCz4NLk for ; Fri, 6 Dec 2024 16:14:05 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=KcqjnkpK; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2a00:1450:4864:20::634 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x634.google.com with SMTP id a640c23a62f3a-aa5ec8d6f64so235925266b.2 for ; Fri, 06 Dec 2024 08:14:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733501644; x=1734106444; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=jSxPp2FWJ8eBRIygmxV90OgcnycOTIQ1c6kLEaydXWY=; b=KcqjnkpK3SxmzC/hWCG7nInDOIblMrfOZRZQvPw56ppplc98B36v9UiqxytoiYe4D/ QVb7lB7BKFHjrUbzFLuFTnawGb8sIL4XJtPLFb0e7Y1l4NHo2OPBdT5tmj7hcFEoBAuw UqeVAkQJA2sVUCj3LhwWBLMCvYguGjN3nIdGsR3RIhH5uW9BslkUSS7cxs09ya/6zuUV OTTosfuXQSTYVjSp/b1GkGzKlcmkHOSh5j7h9g6Ax+5sYIyUSQj23G7y7jEMjoxeO3mw 16R1/hC1izn7vqEfOO9tBZrqyWXuH6wx7XfPXhJIKoCCIn2HE33yfBCeaIuoVGjkfJPQ BfZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733501644; x=1734106444; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=jSxPp2FWJ8eBRIygmxV90OgcnycOTIQ1c6kLEaydXWY=; b=o+YMy6Tog53rYWShil8gs5yIW5NjQsfFyVdoeTOXoyjrLBTRPZz1Glas1E8+7QdrR+ t0cNVlJ1C+GcrldUYgsMrwPg57D62AQi76UBqfQGQCdlHZgWJO2Q7rCB9oWA1ikGQrC/ LkpYQTiMVdXQJ5DaHZ9idPW1+SlBQFF25LY9ybUJnASORznQ5kGOd5ZlZfCLxetBOXYf O2LLJVrxUtvRmUyPffL+ttm4ojKKqRlFzRpp8AD2hwVQAhv6qbyFus7WshvxzfOfKrFe ykochgJBvv8GTUZEF3JoVdiHKvHg0sMc1v/bFQAHE/NWwqFHfAD/Y/Q/gBCYeHf+LDaU RFLw== X-Gm-Message-State: AOJu0YyCmOx+NvToJIqEgywX70zsQy1nFCAJ2/q5OTCJiSprl19i9IUo Rsk+giLTHK6tcY4YwE8G3iBuVAwngfn/K4olehjxPsy5aGIH87XIiiSNZFHLoqdhuDFss9YhPIm pdp+xd4cYmgX1tIPufEbzquKHHRc2 X-Gm-Gg: ASbGnct2nKEzttYSNtcb6HNNYxwgvwU21twPBLu9UV29JnvX9Xngy5/yx9N3gYYRiis AErdtwN4hixAE+st/8ThYCm7vIUk2SVOApTIsGaEr/mOwsBxjT5IXbCSUoSsN5mI= X-Google-Smtp-Source: AGHT+IEfpjX4WpynliGaL6JDws+2Q7eYvZFpBnHTrd3DAaSrLl7lQgDdzC5l1bspmkV/sAjfynq7ZKH8KIop1EnDr/U= X-Received: by 2002:a05:6402:194c:b0:5d0:c697:1f02 with SMTP id 4fb4d7f45d1cf-5d3be6af8ccmr8339366a12.17.1733501643752; Fri, 06 Dec 2024 08:14:03 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 From: Rick Macklem Date: Fri, 6 Dec 2024 08:13:53 -0800 Message-ID: Subject: patch for review To: FreeBSD CURRENT Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-2.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.87)[-0.871]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; NEURAL_HAM_MEDIUM(-0.12)[-0.123]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROMTLD(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; TAGGED_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::634:from] X-Rspamd-Queue-Id: 4Y4bt55XgCz4NLk X-Spamd-Bar: -- Hi, I have put D47743 - Adds a new mountport mount option for NFS D47849 - Man page update for the above on reviews.freebsd.org. I listed "manpages" for review of it, but if anyone would like to take a look at it, please do. I'd particularly like to get the man page patch reviewed. Thanks in advance for looking at this, rick From nobody Fri Dec 6 18:40:02 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y4g6Z46n3z5ggQt; Fri, 06 Dec 2024 18:40:06 +0000 (UTC) (envelope-from SRS0=L9Uk=S7=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y4g6Z1Vxrz4wb4; Fri, 6 Dec 2024 18:40:06 +0000 (UTC) (envelope-from SRS0=L9Uk=S7=klop.ws=ronald-lists@realworks.nl) Authentication-Results: mx1.freebsd.org; none Date: Fri, 6 Dec 2024 19:40:02 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1733510404; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=wo6irttu6YzvMgBq3VfIeBYeAmjweovonVFsAZ24vFg=; b=U+U5Nn05GL3kaCyQ5MF0Krv15x9TrfxRKXAwM8KX2Jvz62ezUOITVH0Z4BdL3YT9oGD2T3 2YYzk8Bbz6cvgmXhgnITDIAKIEezIIu2uqRf1Kgco0DPTywoQbtOu5DIQ7g1GmJZPqjh6s I3hXnypwwAyyKEWVtD54RdsWDj419/caI15l5xvm4+kbNXpk2WUKx1cT+/72uVhQp54Dlk C1eCL8fGczTTt1sV22L4Av1ORMwTtsNe6B0fR45wpxe8UrYcKcTA/i2iEFLrK/TujP/Tvx tJGumj357Jhv4fGPJFgrscg3YYOJzQWIf/k+examNoh81ncvGdRnC4VCNDd/4w== From: Ronald Klop To: FreeBSD User Cc: freebsd-ipfw@freebsd.org, freebsd-current@freebsd.org Message-ID: <279848701.11738.1733510402875@localhost> In-Reply-To: <20241206034709.4dd32cc5@thor.intern.walstatt.dynvpn.de> Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_11737_1079703555.1733510402870" X-Mailer: Realworks (731.101) Importance: Normal X-Priority: 3 (Normal) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL] X-Rspamd-Queue-Id: 4Y4g6Z1Vxrz4wb4 X-Spamd-Bar: ---- ------=_Part_11737_1079703555.1733510402870 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Might be useful to share your ipfw config. Van: FreeBSD User Datum: 6 december 2024 03:47 Aan: freebsd-current@freebsd.org, freebsd-ipfw@freebsd.org Onderwerp: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out > > > Am Thu, 5 Dec 2024 17:33:54 +0100 > FreeBSD User schrieb: > > I found the culprit! > > Disabling IPFW ("ipfw disable firewall") turns system back to normal! > > For the record: on recent CURRENT, since approx. Nov. 30 and/or December 1st CURRENT seems to > corrupt network connections. > > IPFW is compiled statically into the kernel. > > The problem sketched below can be reproduced in a more or less obvious manner on recent > CURRENT: git pull/git clone of a regular FreeBSD source repo or ports via git+https takes > either a couple of time (up to several mintes to initiate the pull) - or, in some worse cases > here, the box runs into > error: RPC failed; curl 56 Recv failure: Operation timed out > > claws-mail complains about "corrupted/broken stream", fetching emails takes Aeons - forever, > the client does not come back even after several hours. > > > On Thu, 5 Dec 2024 16:55:00 +0100 > > Daniel Tameling wrote: > > > > > On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote: > > > > On Wed, 04 Dec 2024 17:20:39 +0000 > > > > "Dave Cottlehuber" wrote: > > > > > > > > Thank you very much for responding! > > > > > > > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: > > > > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem > > > > > > to be stuck > > > > > > forever fetching tarballs from ports, fetching Emails via claws-mail > > > > > > (TLS), opening > > > > > > websites via librewolf and firefox or pulling repositories via git. > > > > > > > > > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > > > > > > 23:11:07 CET 2024 > > > > > > amd64 > > > > > > > > > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: > > > > > > > > > > > > error: RPC failed: curl 56 recv failure: Operation timed out > > > > > > > > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing > > > > > or not, and then working down from there, to tcpdump / wireshark and then if > > > > > necessary into kernel itself. > > > > > > > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and > > > > theory,of course). I tried due to "a feeling" my used older Intel based NIC could have > > > > some checksum issues like in the past (I saw e1000 driver updates recently flowing > > > > into FreeBSD CURRENT). > > > > > > > > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball, > > > > > and try it again in curl, with tcpdump already prepared to capture traffic to the > > > > > remote host. > > > > > > > > > > tcpdump -w /tmp/curl.pcap -i ... host ... > > > > > > > > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace > > > > > /tmp/curl.log https://what.ev/er > > > > > > > > > > I would guess that between the two something useful should pop up. > > > > > > > > > > I like opening the pcap in wireshark, it often has angry red and black highlighted > > > > > lines already giving me a hint. > > > > > > > > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS > > > > > traffic as well in case there are issues further in. Very handy, > > > > > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. > > > > > > > > > > If your issues only occur with git pull, its also curl inside and supports similar > > > > > debugging. Ferreting > > > > > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. > > > > > > > > > > A+ > > > > > Dave > > > > > > > > > > > > > Thanks for the hints and precious tips! I'll digg deeper into the matter. > > > > > > > > In the meanwhile, I updated some other machines running CURRENT since approx. two > > > > weeks with an older CURRENT to the most recent one - and face similar but not > > > > identical problems! > > > > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems > > > > except they take longer to accomplish than expected. > > > > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout. > > > > > > > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD > > > > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), cloning > > > > attempts for 14.2-RELENG ends up in this mess: > > > > > > > > # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/ > > > > Cloning into '14.2-RELENG/src'... > > > > error: RPC failed; curl 56 Recv failure: Operation timed out > > > > fatal: expected 'packfile' > > > > > > > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's > > > > kernel is very similar to the box I reported the issue first time, both do have > > > > customized kernels (in most cases, I compile several modules like ZFS and > > > > several NETGRAPH modules statically into the kernel - a habit inherited from a small > > > > FBSD project I configured (I wouldn't say developed) which does not allow loadable > > > > kernel modules due to regulations. > > > > > > > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the > > > > phenomena and its distribution over a bunch of CURRENT boxes with different OS states > > > > seemingly show different behviour. > > > > > > > > And for the record: I also build my ports via poudriere and mostly via make. I also > > > > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and > > > > so on to ensure having latest sources/packages. > > > > > > > > (I repeat myself here again, sorry, its for the record). > > > > > > > > Will report in on further development and "investigations" > > > > > > > > Kind regards and thanks, > > > > > > > > oh > > > > > > > > > > > > > > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some > > > networking issues that got fixed later. > > > > No, pure Hardware and FreeBSD ... > > > > > > > > Otherwise I would start with ping and traceroute to figure out if they show this issue > > > and where it occurs. > > > > > > > > > > > -- > O. Hartmann > > > > > ------=_Part_11737_1079703555.1733510402870 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit Might be useful to share your ipfw config. 

Van: FreeBSD User <freebsd@walstatt-de.de>
Datum: 6 december 2024 03:47
Aan: freebsd-current@freebsd.org, freebsd-ipfw@freebsd.org
Onderwerp: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out

Am Thu, 5 Dec 2024 17:33:54 +0100
FreeBSD User schrieb:

I found the culprit!

Disabling IPFW ("ipfw disable firewall") turns system back to normal!

For the record: on recent CURRENT, since approx. Nov. 30 and/or December 1st CURRENT seems to
corrupt network connections.

IPFW is compiled statically into the kernel.

The problem sketched below can be reproduced in a more or less obvious manner on recent
CURRENT: git pull/git clone of a regular FreeBSD source repo or ports via git+https takes
either a couple of time (up to several mintes to initiate the pull) - or, in some worse cases
here, the box runs into
error: RPC failed; curl 56 Recv failure: Operation timed out

claws-mail complains about "corrupted/broken stream", fetching emails takes Aeons - forever,
the client does not come back even after several hours.

> On Thu, 5 Dec 2024 16:55:00 +0100
> Daniel Tameling wrote:
>
> > On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote:  
> > > On Wed, 04 Dec 2024 17:20:39 +0000
> > > "Dave Cottlehuber" wrote:
> > >
> > > Thank you very much for responding!
> > >     
> > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote:    
> > > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem
> > > > > to be stuck
> > > > > forever fetching tarballs from ports, fetching Emails via claws-mail
> > > > > (TLS), opening
> > > > > websites via librewolf and firefox or pulling repositories via git.
> > > > >
> > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec  2
> > > > > 23:11:07 CET 2024
> > > > > amd64
> > > > >
> > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes:
> > > > >
> > > > > error: RPC failed: curl 56 recv failure: Operation timed out      
> > > >
> > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing
> > > > or not, and then working down from there, to tcpdump / wireshark and then if
> > > > necessary into kernel itself.    
> > >
> > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and
> > > theory,of course). I tried due to "a feeling" my used older Intel based NIC could have
> > > some checksum issues like in the past (I saw e1000 driver updates recently flowing
> > > into FreeBSD CURRENT).    
> > > >
> > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball,
> > > > and try it again in curl, with tcpdump already prepared to capture traffic to the
> > > > remote host.
> > > >
> > > > tcpdump -w /tmp/curl.pcap -i ... host ...
> > > >
> > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace
> > > > /tmp/curl.log https://what.ev/er
> > > >
> > > > I would guess that between the two something useful should pop up.
> > > >
> > > > I like opening the pcap in wireshark, it often has angry red and black highlighted
> > > > lines already giving me a hint.
> > > >
> > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS
> > > > traffic as well in case there are issues further in. Very handy,
> > > > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that.
> > > >
> > > > If your issues only occur with git pull, its also curl inside and supports similar
> > > > debugging. Ferreting
> > > > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info.
> > > >
> > > > A+
> > > > Dave
> > > >     
> > >
> > > Thanks for the hints and precious tips! I'll digg deeper into the matter.
> > >
> > > In the meanwhile, I updated some other machines running CURRENT since approx. two
> > > weeks with an older CURRENT to the most recent one - and face similar but not
> > > identical problems!
> > > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems
> > > except they take longer to accomplish than expected.
> > > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout.
> > >
> > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD
> > > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec  4 22:22:22 CET 2024 amd64), cloning
> > > attempts for 14.2-RELENG ends up in this mess:
> > >
> > > # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/
> > > Cloning into '14.2-RELENG/src'...
> > > error: RPC failed; curl 56 Recv failure: Operation timed out
> > > fatal: expected 'packfile'
> > >
> > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's
> > > kernel is very similar to the box I reported the issue first time, both do have
> > > customized kernels (in most cases, I compile several modules like ZFS and
> > > several NETGRAPH modules statically into the kernel - a habit inherited from a small
> > > FBSD project I configured (I wouldn't say developed) which does not allow loadable
> > > kernel modules due to regulations.
> > >
> > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the
> > > phenomena and its distribution over a bunch of CURRENT boxes with different OS states
> > > seemingly show different behviour.
> > >
> > > And for the record: I also build my ports via poudriere and mostly via make. I also
> > > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and
> > > so on to ensure having latest sources/packages.
> > >
> > > (I repeat myself here again, sorry, its for the record).
> > >
> > > Will report in on further development and "investigations"
> > >
> > > Kind regards and thanks,
> > >
> > > oh
> > >
> > >     
> >
> > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some
> > networking issues that got fixed later.  
>
> No, pure Hardware and FreeBSD ...
>
> >
> > Otherwise I would start with ping and traceroute to figure out if they show this issue
> > and where it occurs.
> >   
>
>



-- 
O. Hartmann




------=_Part_11737_1079703555.1733510402870-- From nobody Fri Dec 6 20:09:20 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y4j691dMQz5fp8J; Fri, 06 Dec 2024 20:09:53 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp052.goneo.de (smtp052.goneo.de [85.220.129.60]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y4j685hScz564g; Fri, 6 Dec 2024 20:09:52 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Authentication-Results: mx1.freebsd.org; none Received: from hub1.goneo.de (hub1.goneo.de [85.220.129.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id 5D544240D18; Fri, 6 Dec 2024 21:09:50 +0100 (CET) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id 74DE2240562; Fri, 6 Dec 2024 21:09:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1733515788; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9tlVOqP5LtBTwqnsT9G2I7CBMXXcRs9yIPWa6+WZImA=; b=lczPX0ODEy7JGetelVShwKORM1oNWwtcFbgfBKY/66Jf5pwnIzRP//E8EJSF43nfyXomtm hdE7DFTVFf6T2GtwJeLgiEZx3DEVooNOGpbcBdItGMgJBgjQZNL0izTtT/gOTVlTPDQcIO R7juuYScoOkIwleOLHS45QGLFHAanTuIHhmszh4hbsl/lFWsAUjOOmZxKJiFxkeDdTl/zQ 6XP+CayFVTRhOq2qTiI5ZfBePhSZV7psBs/DK1i4wWTayQG4ODbf5xBkXlQa201/7FH9y5 cCknCbX4soTwswiSC6AHez/Tk7XhXxGtTy7Z4cniCCuKN139sFp58TyBcNHB5Q== Received: from thor.intern.walstatt.dynvpn.de (dynamic-089-012-099-016.89.12.pool.telefonica.de [89.12.99.16]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id 2EC8C240560; Fri, 6 Dec 2024 21:09:48 +0100 (CET) Date: Fri, 6 Dec 2024 21:09:20 +0100 From: FreeBSD User To: Ronald Klop Cc: freebsd-ipfw@freebsd.org, freebsd-current@freebsd.org Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: <20241206210947.3ae835e4@thor.intern.walstatt.dynvpn.de> In-Reply-To: <279848701.11738.1733510402875@localhost> References: <20241206034709.4dd32cc5@thor.intern.walstatt.dynvpn.de> <279848701.11738.1733510402875@localhost> Organization: walstatt-de.de List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: 51ce02 X-Rspamd-UID: 21f4eb X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE] X-Rspamd-Queue-Id: 4Y4j685hScz564g X-Spamd-Bar: ---- Am Fri, 6 Dec 2024 19:40:02 +0100 (CET) Ronald Klop schrieb: > Might be useful to share your ipfw config. Sorry, my posting must have been disturbing (having in mind a "deny any rule and then disabling the FW ...). Well, the IPFW setup itself is explained quickly - I use almost the vanilla rc.conf-issued IPFW (settings: firewall_type="workstation", firewall_logif="YES", firewall_myservices="22/tcp", firewall_allowservices="any"). The hosts in question have the following kernel configuration, I provide the option tags that might be of interest or, if not, just for the record, as they are not part of GENERIC, see below. Also, I'll provide some sysctl setting performed via /etc/sysctl.conf.local, see below. The configuration and settings have been mostly unchanged over a couple of months for now and did not induce trouble so far. As it deemed fit regarding time and my limited skills, I disabled and enabled piece by piece of the MAC_ and NETGRAPH_ options - without any success so far - my "measurement" is fetching emails via claws-mail (all TLS). claws-mail reports "corrupted/broken stream", does have authetication issues and is de facto unusable - it doesn't refresh IMAP based email fetches and doesn't even quit without a hard kill. Another "indicator" is the time taken to "git pull" of ZFS filesystems: cloning and pulling takes unusual long (/usr/src is UFS/FFS, /usr/ports on a ZFS pool and since the problem occured, it makes a mutual difference). While git pull or clone mutually stuck and claws-mail is endlessly fetching/authenticating emails and never responding back in a usable manner, performing "ipfw disable firewall" makes all of a sudden the system work again as usual and expected. As reported - the problem spreads across all of my CURRENT hosts as I'm going to update them towards a recent CURRENT (they all share similar static kernel configs as described here). Most of the boxes do not show the weird reluctant behaviour when pulling via git, but weren't capable of cloning, bailing out with the timeout reported earlier. I use one CURRENT box as my personal desktop, so no other (server) CURRENT show the Email problem in detail as described. And, for the record: I haven't commented out the "options IPFIREWALL" yet in the kernel config ... Kind regards oh [ KERNEL config different from vanilla GENERIC ] options RATELIMIT options ZFS options TCPHPTS options MROUTING options IPSEC options SCTP options MAC_BSDEXTENDED options MAC_PORTACL options MAC_IPACL options MAC_NTPD #options MAC_DO options NETGRAPH options NETGRAPH_IPFW options NETGRAPH_ETHER options NETGRAPH_EIFACE options NETGRAPH_VLAN #options NETGRAPH_NAT options NETGRAPH_DEVICE #options NETGRAPH_PPPOE options NETGRAPH_SOCKET options NETGRAPH_KSOCKET options NETGRAPH_NETFLOW #options NETGRAPH_CAR # IPFW firewall options IPFIREWALL options IPFIREWALL_VERBOSE options DUMMYNET # traffic shaper options BPF_JITTER # adds support for BPF just-in-time compiler. # Pseudo devices not in GENERIC. device enc # IPsec device device stf # 6to4 IPv6 over IPv4 encapsulation device carp # Common address redundancy protocol device lagg # Link aggregation device gre # GRE Tunnel device epair # A pair of virtual back-to-back connected Ethernet interfaces device if_bridge # bridge device device vxlan # Virtual eXtensible LAN interface For the MAC_ Modules: the appropriate OIDs (sysctl) are disabled as far as the MAC module influence the initial behaviour if unconfigured, for instance (/etc/sysctl.conf.local) [ /etc/sysctl.conf.local ] security.mac.bsdextended.enabled=0 security.mac.mls.enabled=0 security.mac.portacl.enabled=0 security.mac.do.enabled=0 security.mac.ipacl.ipv6=0 security.mac.ipacl.ipv4=0 # net.bpf.optimize_writers=1 # net.inet.ip.fw.verbose=1 #net.inet.ip.fw.verbose_limit=10 net.inet.ip.fw.dyn_keep_states=1 > > Van: FreeBSD User > Datum: 6 december 2024 03:47 > Aan: freebsd-current@freebsd.org, freebsd-ipfw@freebsd.org > Onderwerp: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: > Operation timed out > > > > > > > Am Thu, 5 Dec 2024 17:33:54 +0100 > > FreeBSD User schrieb: > > > > I found the culprit! > > > > Disabling IPFW ("ipfw disable firewall") turns system back to normal! > > > > For the record: on recent CURRENT, since approx. Nov. 30 and/or December 1st CURRENT seems > > to corrupt network connections. > > > > IPFW is compiled statically into the kernel. > > > > The problem sketched below can be reproduced in a more or less obvious manner on recent > > CURRENT: git pull/git clone of a regular FreeBSD source repo or ports via git+https takes > > either a couple of time (up to several mintes to initiate the pull) - or, in some worse > > cases here, the box runs into > > error: RPC failed; curl 56 Recv failure: Operation timed out > > > > claws-mail complains about "corrupted/broken stream", fetching emails takes Aeons - > > forever, the client does not come back even after several hours. > > > > > On Thu, 5 Dec 2024 16:55:00 +0100 > > > Daniel Tameling wrote: > > > > > > > On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote: > > > > > On Wed, 04 Dec 2024 17:20:39 +0000 > > > > > "Dave Cottlehuber" wrote: > > > > > > > > > > Thank you very much for responding! > > > > > > > > > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: > > > > > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem > > > > > > > to be stuck > > > > > > > forever fetching tarballs from ports, fetching Emails via claws-mail > > > > > > > (TLS), opening > > > > > > > websites via librewolf and firefox or pulling repositories via git. > > > > > > > > > > > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > > > > > > > 23:11:07 CET 2024 > > > > > > > amd64 > > > > > > > > > > > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: > > > > > > > > > > > > > > error: RPC failed: curl 56 recv failure: Operation timed out > > > > > > > > > > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing > > > > > > or not, and then working down from there, to tcpdump / wireshark and then if > > > > > > necessary into kernel itself. > > > > > > > > > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and > > > > > theory,of course). I tried due to "a feeling" my used older Intel based NIC could > > > > > have some checksum issues like in the past (I saw e1000 driver updates recently > > > > > flowing into FreeBSD CURRENT). > > > > > > > > > > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable > > > > > > tarball, and try it again in curl, with tcpdump already prepared to capture > > > > > > traffic to the remote host. > > > > > > > > > > > > tcpdump -w /tmp/curl.pcap -i ... host ... > > > > > > > > > > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace > > > > > > /tmp/curl.log https://what.ev/er > > > > > > > > > > > > I would guess that between the two something useful should pop up. > > > > > > > > > > > > I like opening the pcap in wireshark, it often has angry red and black highlighted > > > > > > lines already giving me a hint. > > > > > > > > > > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS > > > > > > traffic as well in case there are issues further in. Very handy, > > > > > > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. > > > > > > > > > > > > If your issues only occur with git pull, its also curl inside and supports similar > > > > > > debugging. Ferreting > > > > > > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. > > > > > > > > > > > > A+ > > > > > > Dave > > > > > > > > > > > > > > > > Thanks for the hints and precious tips! I'll digg deeper into the matter. > > > > > > > > > > In the meanwhile, I updated some other machines running CURRENT since approx. two > > > > > weeks with an older CURRENT to the most recent one - and face similar but not > > > > > identical problems! > > > > > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems > > > > > except they take longer to accomplish than expected. > > > > > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout. > > > > > > > > > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD > > > > > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), > > > > > cloning attempts for 14.2-RELENG ends up in this mess: > > > > > > > > > > # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/ > > > > > Cloning into '14.2-RELENG/src'... > > > > > error: RPC failed; curl 56 Recv failure: Operation timed out > > > > > fatal: expected 'packfile' > > > > > > > > > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's > > > > > kernel is very similar to the box I reported the issue first time, both do have > > > > > customized kernels (in most cases, I compile several modules like ZFS and > > > > > several NETGRAPH modules statically into the kernel - a habit inherited from a small > > > > > FBSD project I configured (I wouldn't say developed) which does not allow loadable > > > > > kernel modules due to regulations. > > > > > > > > > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the > > > > > phenomena and its distribution over a bunch of CURRENT boxes with different OS states > > > > > seemingly show different behviour. > > > > > > > > > > And for the record: I also build my ports via poudriere and mostly via make. I also > > > > > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and > > > > > so on to ensure having latest sources/packages. > > > > > > > > > > (I repeat myself here again, sorry, its for the record). > > > > > > > > > > Will report in on further development and "investigations" > > > > > > > > > > Kind regards and thanks, > > > > > > > > > > oh > > > > > > > > > > > > > > > > > > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some > > > > networking issues that got fixed later. > > > > > > No, pure Hardware and FreeBSD ... > > > > > > > > > > > Otherwise I would start with ping and traceroute to figure out if they show this issue > > > > and where it occurs. > > > > > > > > > > > > > > > > > > -- > > O. Hartmann > > > > > > > > > > -- O. Hartmann From nobody Sun Dec 8 16:03:10 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y5qXn0fX4z5gq6r for ; Sun, 08 Dec 2024 16:03:21 +0000 (UTC) (envelope-from minsoochoo0122@proton.me) Received: from mail-4027.protonmail.ch (mail-4027.protonmail.ch [185.70.40.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y5qXl4wtsz54kx for ; Sun, 8 Dec 2024 16:03:19 +0000 (UTC) (envelope-from minsoochoo0122@proton.me) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=proton.me header.s=rjh5q7l25rdd7a44gyrs6he3ym.protonmail header.b=E+sw8vYF; spf=pass (mx1.freebsd.org: domain of minsoochoo0122@proton.me designates 185.70.40.27 as permitted sender) smtp.mailfrom=minsoochoo0122@proton.me; dmarc=pass (policy=quarantine) header.from=proton.me DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proton.me; s=rjh5q7l25rdd7a44gyrs6he3ym.protonmail; t=1733673795; x=1733932995; bh=R9j6LzTzO3Gepw9FJCxcp4lHt0Jdni/bWKviATajJ5o=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector:List-Unsubscribe:List-Unsubscribe-Post; b=E+sw8vYFzOdIperQM1cFTczU+3Inedp1cFyg5fXSt/u7/OqKfv7241EC5nxU73mRn f44/IYOYPTNAlw6p8D424YL3RW+rDZ3pCBjkvjP2QO4Cv3l4akfoZSysoa4b7oKZN5 asA7gCY0du39gTOWlUDK1FwTXMVMZ/2v7ep8xIogWpwfhoR/YEgbE+xPqt7iT7dDj2 AkL8COD0L0YOTpdnv5q09TM8mgu7D6FaWdNPcn1o7tbvbTHVY3N0m4EIxf8RCA8VWn 9h9fwQJCX+iwrHBRV+I2XcXfyNn75QZiZhQ5VwLlVVL8IfvnSZYYZhZH3aC8HhEzlf jhJsn7PSZlv+g== Date: Sun, 08 Dec 2024 16:03:10 +0000 To: Warner Losh From: Minsoo Choo Cc: cglogic , FreeBSD CURRENT Subject: Re: Long time outdated jemalloc Message-ID: <4L_wVuJx1yIMEv85fQKvrJp8QiaTK4Fe_TvymIq0vcdwdHqa06Ys4lqAM8aHb-kefxPiIZW7kxT8qI7hmv4bLngKUlzIWfBVzDcaz4VRIPY=@proton.me> In-Reply-To: References: Feedback-ID: 45891198:user:proton X-Pm-Message-ID: 2fa7e14fb8bdcbf798ba4ba21ee3c781855d9c8c List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1=_LL69nXigG2smgSsejGdXqg0i84csaQ4J6T0lWPRUQ" X-Spamd-Result: default: False [-3.90 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[proton.me,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:185.70.40.0/24]; R_DKIM_ALLOW(-0.20)[proton.me:s=rjh5q7l25rdd7a44gyrs6he3ym.protonmail]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_BASE64_TEXT(0.10)[]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[185.70.40.27:from]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:62371, ipnet:185.70.40.0/24, country:CH]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[protonmail.com,freebsd.org]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DKIM_TRACE(0.00)[proton.me:+] X-Rspamd-Queue-Id: 4Y5qXl4wtsz54kx X-Spamd-Bar: --- --b1=_LL69nXigG2smgSsejGdXqg0i84csaQ4J6T0lWPRUQ Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 SSByZXNvbHZlZCBtZXJnZSBjb25mbGljdCBieSByZWJhc2luZyBtYWluLiBXaGF0J3MgbmV4dD8K Cmh0dHBzOi8vZ2l0aHViLmNvbS9mcmVlYnNkL2ZyZWVic2Qtc3JjL3B1bGwvMTMzNwpPbiBTdW5k YXksIERlY2VtYmVyIDFzdCwgMjAyNCBhdCAyOjA1IEFNLCBXYXJuZXIgTG9zaCA8aW1wQGJzZGlt cC5jb20+IHdyb3RlOgoKPiAoc29ycnkgdG8gZm9sbG93IHVwIHRvIG15IG93biBlbWFpbCBhbmQg dG9wcG9zdGluZykKPgo+IEkgZ290IHRoZSB2ZW5kb3IgYnJhbmNoIGJvb3RzdHJhcHBlZDogSSBj cmVhdGVkIHZlbmRvci9qZW1hbGxvYyBhbmQgdGFnZ2VkIHZlbmRvci9qZW1hbGxvYy81LjIuMSwK PiBhbmQgY3JlYXRlZCBhIG1lcmdlIGNvbW1pdCBmcm9tIHRoYXQgYnJhbmNoIHRvIG1haW4uIEkg aGFkIHRvIHR3ZWFrIHRoZQo+IEZSRUVCU0QtWGxpc3QgYSBsaXR0bGUuSSd2ZSBub3QgdXBkYXRl ZCB0aGUgb3RoZXIgdHdvIEZSRUVCU0QtKiBmaWxlcywgYnV0IHRoZSBzdGVwcyB3ZXJlCj4gZG9j dW1lbnRlZCBpbiB0aGUgY29tbWl0IG1lc3NhZ2VzICh0aGUgdmVuZG9yIGJyYW5jaCBvbmUgaXMg ZXNwZWNpYWxseSBsb25nIGFuZAo+IGxpa2VseSBzaG91bGQgbWlncmF0ZSBpbnRvIHRoZSBkZXZl bG9wZXIgaGFuZGJvb2spLiBGUkVFQlNELXVwZGF0ZSBpcyBiYXNpY2FsbHkKPiBhIHNoZWxsIHNj cmlwdCB0byBkbyB0aGUgc2FtZSB0aGluZyB0aGF0IGdpdCBzdWJ0cmVlIG1lcmdlIGRvZXMsIHRo b3VnaCBJJ20gc3VyZSBzb21lCj4gdHdlYWtzIGNvdWxkIGhlbHAgdGhlIG5leHQgdGltZSAoaWYg dGhlcmUgaXMgYSBuZXh0IHRpbWUsIGplbWFsbG9jIHVwc3RyZWFtIHNlZW1zIHRvCj4gaGF2ZSBz bG93ZWQgd2F5IGRvd24gb2YgbGF0ZSkuCj4KPiBOZXh0IHVwLEknbGwgY3JlYXRlIDUuMy4wIGlt cG9ydCBvbiB0aGUgdmVuZG9yIGJyYW5jaCwgZG8gdGhlIG1lcmdlIGFuZCBzdGFydCB0ZXN0aW5n IChpdCB3aWxsIGJlCj4gTWluc29vJ3MgcHVsbCByZXF1ZXN0LCByZWJhc2VkLCB3aXRoIGFueSBj b25mbGljdHMgcmVzb2x2ZWQgYW5kIG1lcmdlIGNvbW1pdCByZWNvcmRlZCkuCj4gQnV0IHRoYXQg d2lsbCBoYXZlIHRvIGJlIHRvbW9ycm93IG9yIG1vcmUgbGlrZWx5IGR1cmluZyB0aGUgd29yayB3 ZWVrLiBJJ20gdG9vIHRpcmVkIHRvbmlnaHQKPiB0byBnZXQgaXQgcmlnaHQgYXQgdGhlIG1vbWVu dC4KPgo+IEFuZCBhIHNwZWNpYWwgdGhhbmtzIHRvIGVtYXN0ZSBmb3IgZ2l2aW5nIGJ6IHRoZSBy aWdodCByZWNpcGUgZm9yIGRvaW5nIHRoZSBzdWJ0cmVlIG1lcmdlCj4gdy9vIGdpdCBzdWJ0cmVl IGZvciBoaXMgd29yayBvbiB0aGUgbGludXggd2lmaSBkcml2ZXJzIGluIHRoZSB0cmVlLgo+Cj4g V2FybmVyCj4KPiBPbiBTYXQsIE5vdiAzMCwgMjAyNCBhdCA5OjM44oCvUE0gV2FybmVyIExvc2gg PGltcEBic2RpbXAuY29tPiB3cm90ZToKPgo+PiBZZWEsIEkgbmVlZCB0byBnZXQgYSBjb3B5IG9m IGplbWFsbG9jIDUuMy4wIGFuZCA1LjIuMSB0byB0cnkgdG8gJ2Jvb3RzdHJhcCcgdGhlIHZlbmRv ciBicmFuY2guCj4+IFRoZW4gSSBuZWVkIHRvIGJvb3RzdHJhcCBpdC4uLgo+Pgo+PiBJIGp1c3Qg ZGlkIHRoZSBzYW1lIHdpdGggZWRrMiAod2hpY2ggaGFkIGEgdmVuZG9yIGJyYW5jaCwgYnV0IGhh ZG4ndCBiZWVuIHVwZGF0ZWQgc2luY2Ugc3ZuIHRpbWVzKS4KPj4gSG93ZXZlciwgamVtYWxsb2Mg ZG9lc24ndCBoYXZlIGEgdmVuZG9yIGJyYW5jaCB5ZXQsIHNvIEknbGwgaGF2ZSB0byBjcmVhdGUg dGhhdCwgYnV0IEknbGwgc3RhcnQgd2l0aCB0aGUKPj4gY3VycmVudCB2ZXJzaW9uIHJhdGhlciB0 aGFuIGRvaW5nIGZ1bGwgaGlzdG9yeS4uLiBTbyBJJ2xsIHN0YXJ0IHRoZXJlLgo+PiBJIGFsc28g anVzdCBkaWQgYXdrIGFuZCBsdWEsIHNvIG9uY2UgSSBoYXZlIHRoaW5ncyBib290c3RyYXBwZWQs IEknbGwgYmUgYWJsZSB0byBhZGQgNS4zLjAgYW5kIHRoZW4gbGF5ZXIgTWluc29vJ3Mgb24KPj4g dG9wIG9mIHRoYXQgYW5kIHRoZW4gc3RhcnQgdGVzdGluZyBpdCBzb21laG93Lgo+Pgo+PiBNYWxs b2MgbWFrZXMgbWUgbmVydm91cyB0byB0b3VjaCwgaG9uZXN0bHksIGJ1dCBJJ2xsIGdpdmUgaXQg YSBnbyBhbmQgdGVzdCBib290IG9uIG15IHN5c3RlbSBhbmQKPj4gbWF5YmUgc2VlIGlmIHdlIGNh biBzdXJ2aXZlIGEgd29ya2xvYWQgYXQgd29yayB3L28gcmVncmVzc2lvbnMuLi4gQnV0IEkgY2Fu J3QgZG8gYSBmdWxsIHRlc3Qgd2l0aCBsb3RzCj4+IG9mIG1hY2hpbmVzIHVudGlsIGFmdGVyIHRo ZSBmaXJzdCBvZiB0aGUgeWVhciAodGhvdWdoIEkgY2FuIGRvIGEgY291cGxlIGZvciBhIGZldyBk YXlzIGJlZm9yZSB0aGVuKS4KPj4KPj4gU28gbXkgbmV4dCBzdGVwIGlzIHRvIGJvb3RzdHJhcCB0 aGUgdmVuZG9yIGJyYW5jaC4uLiBJJ2xsIGdpdmUgdGhhdCBhIHRyeSB0b25pZ2h0Lgo+Pgo+PiBX YXJuZXIKPj4KPj4gT24gU2F0LCBOb3YgMzAsIDIwMjQgYXQgODoyNuKAr1BNIE1pbnNvbyBDaG9v IDxtaW5zb29jaG9vMDEyMkBwcm90b24ubWU+IHdyb3RlOgo+Pgo+Pj4gSSBoYXZlIGFscmVhZHkg c3VibWl0dGVkIFBSIG9uIGdpdGh1YiAoaHR0cHM6Ly9naXRodWIuY29tL2ZyZWVic2QvZnJlZWJz ZC1zcmMvcHVsbC8xMzM3KSBhbmQgcGhhYnJpY2F0b3IgKGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNk Lm9yZy9ENDE0MjEpLiBJIGRvbid0IGhhdmUgYWNjZXNzIChjb21taXQgYml0KSB0byBmcmVlYnNk IGdpdCByZXBvLCBzbyB0aGVyZSBpcyBub3RoaW5nIEkgY2FuIGRvIGF0IHRoaXMgcG9pbnQgc2lu Y2UgdmVuZG9yIGltcG9ydCBhbmQgbGFuZGluZyBwYXRjaGVzIHJlcXVpcmVzIGNvbW1pdCBiaXQu Cj4+PiBPbiBTYXR1cmRheSwgTm92ZW1iZXIgMzB0aCwgMjAyNCBhdCAxOjQyIFBNLCBjZ2xvZ2lj IDxjZ2xvZ2ljQHByb3Rvbm1haWwuY29tPiB3cm90ZToKPj4+Cj4+Pj4gSSBzZWUsIGl0IGhhcHBl bnMuCj4+Pj4gTWF5YmUgYW5vdGhlciBjb21taXR0ZXIgd2lsbCB2b2x1bnRlZXIgdG8gZG8gdGhl IHVwZGF0ZS4KPj4+PiBJIGhvcGUgaXQgd2lsbCBtYWtlIGl0cyB3YXkgaW50byAxNS4wIHJlbGVh c2UuCj4+Pj4KPj4+PiBUaGFua3MuCj4+Pj4gT24gRnJpZGF5LCBOb3ZlbWJlciAyOXRoLCAyMDI0 IGF0IDk6MzggUE0sIFdhcm5lciBMb3NoIDxpbXBAYnNkaW1wLmNvbT4gd3JvdGU6Cj4+Pj4KPj4+ Pj4gSSd2ZSBiZWVuIHN3YW1wZWQuIHdlIG5lZWQgdG8gYm9vdHN0cmFwIHRoZSB2ZW5kb3IgYnJh bmNoLCBhbmQgdGhlIHdheSBwcmlvciB1cGRhdGVzIHdlcmUgZG9uZQo+Pj4+PiBpc24ndCBzbyBn cmVhdC4KPj4+Pj4KPj4+Pj4gV2FybmVyCj4+Pj4+Cj4+Pj4+IE9uIE1vbiwgTm92IDI1LCAyMDI0 IGF0IDI6MjHigK9BTSBjZ2xvZ2ljIDxjZ2xvZ2ljQHByb3Rvbm1haWwuY29tPiB3cm90ZToKPj4+ Pj4KPj4+Pj4+IEhlbGxvIGd1eXMsCj4+Pj4+Pgo+Pj4+Pj4gSG93IHRoZSB1cGRhdGUgb2YgamVt YWxsb2MgaXMgZ29pbmc/IEl0J3MgTm92ZW1iZXIgbm93Lgo+Pj4+Pj4KPj4+Pj4+IFRoYW5rcy4K Pj4+Pj4+IE9uIE1vbmRheSwgSnVseSAyMm5kLCAyMDI0IGF0IDc6MDIgUE0sIE1pbnNvbyBDaG9v IDxtaW5zb29jaG9vMDEyMkBwcm90b24ubWU+IHdyb3RlOgo+Pj4+Pj4KPj4+Pj4+PiBGaXJzdCwg c29ycnkgZm9yIGxhdGUgcmVzcG9uc2UuCj4+Pj4+Pj4KPj4+Pj4+PiBjZ2xvZ2ljLCB0aGFuayB5 b3UgZm9yIGJyaW5naW5nIHVwIHRoaXMgaXNzdWUgYWdhaW4gc2luY2UgSSBuZWFybHkgZm9yZ290 IHRoYXQgdGhpcyBpc3N1ZSB3YXMgc3RpbGwgb3Blbi4KPj4+Pj4+Pgo+Pj4+Pj4+IFdhcm5lciwg YXMgSSBjYW4ndCBhY2Nlc3MgdG8gbXkgRnJlZUJTRCBpbnN0YW5jZSB1bnRpbCB0aGUgZW5kIG9m IEF1Z3VzdCwgYnV0IEkgY2FuIHN0aWxsIGVkaXQgYW5kIHB1c2ggdGhlIGNvZGUgdGhyb3VnaCBt eSBBcm0gTWFjLiBUaGlzIG1lYW5zIHRoYXQgSSBjYW4ndCB0ZXN0IHRoZSB1cGRhdGVkIGNvZGUg b24gbXkgbWFjaGluZSwgYnV0IEkgY2FuIGpvaW4gdGhlIHJldmlldyBwcm9jZXNzIGFuZCBsaXN0 ZW4gdG8gY2hhbmdlIHByb3Bvc2Fscy4KPj4+Pj4+Pgo+Pj4+Pj4+IEknbGwgb3BlbiBhIEdpdGh1 YiBQUiBpbiBhIGZldyBob3Vycy4gKFRoZSBwaGFicmljYXRvciByZXZpZXcgd2lsbCBzdGF5IG9w ZW5lZCBqdXN0IGluIGNhc2UpCj4+Pj4+Pj4gT24gTW9uZGF5LCBKdWx5IDIybmQsIDIwMjQgYXQg NTowOCBBTSwgV2FybmVyIExvc2ggPGltcEBic2RpbXAuY29tPiB3cm90ZToKPj4+Pj4+Pgo+Pj4+ Pj4+PiBPbiBTdW4sIEp1bCAyMSwgMjAyNCBhdCAyOjAz4oCvUE0gY2dsb2dpYyA8Y2dsb2dpY0Bw cm90b25tYWlsLmNvbT4gd3JvdGU6Cj4+Pj4+Pj4+Cj4+Pj4+Pj4+PiBPbiBTdW5kYXksIEp1bHkg MjFzdCwgMjAyNCBhdCA2OjU0IEFNLCBXYXJuZXIgTG9zaCA8aW1wQGJzZGltcC5jb20+IHdyb3Rl Ogo+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+PiBPbiBTYXQsIEp1bCAyMCwgMjAyNCBhdCAxOjU54oCvQU0g Y2dsb2dpYyA8Y2dsb2dpY0Bwcm90b25tYWlsLmNvbT4gd3JvdGU6Cj4+Pj4+Pj4+Pj4KPj4+Pj4+ Pj4+Pj4gSGVsbG8gRnJlZUJTRCBjb21tdW5pdHksCj4+Pj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pj4+IEFm dGVyIEphc29uIEV2YW5zIHN0ZXBwZWQgYXNpZGUgZnJvbSBtYWludGFpbmluZyBqZW1hbGxvYyBp biBGcmVlQlNELCBpdCdzIG5vdCB1cGRhdGluZyBpbiB0aW1lIGFueW1vcmUuCj4+Pj4+Pj4+Pj4+ IFZlcnNpb24gNS4zLjAgd2FzIHJlbGVhc2VkIE1heSA2LCAyMDIyIGFuZCBGcmVlQlNEIHN0aWxs IG5vdCBpbXBvcnRlZCBpdCBpbnRvIHRoZSB0cmVlLgo+Pj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4+PiBU aGVyZSBpcyBhIHBlbmRpbmcgcmV2aWV3IGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9ENDE0 MjEgZnJvbSBBdWcgMTEsIDIwMjMuCj4+Pj4+Pj4+Pj4+IEknbSBzdWNjZXNzZnVsbHkgcnVubmlu ZyBGcmVlQlNEL2FtZDY0IHN5c3RlbSB3aXRoIEQ0MTQyMSBhcHBsaWVkIGZvciA4IG1vbnRocywg YXMgd2VsbCBhcyBtYW55IG90aGVyIHBlb3BsZS4KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gQ2Fu IGl0IGJlIHJldmlld2VkIGFuZCBjb21taXR0ZWQgdG8gQ1VSUkVOVD8KPj4+Pj4+Pj4+Pj4gT3Is IGlmIHRoZXJlIGlzIG5vIGNvbW1pdHRlcnMgd2lsbGluZyB0byBkbyBpdCwgY2FuIGNvbW1pdCBi aXQgYmUgZ2l2ZW4gdG8gc3VibWl0dGVyIG9yIGFub3RoZXIgcGVyc29uIHdpbGxpbmcgdG8gZG8g dGhpcz8KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+Pj4gSXQncyB2ZXJ5IGRpc2FwcG9pbnRpbmcgd2hl biB1c2VycyBzcGVuZCB0aGVpciB0aW1lIHRvIGZpbGwgc3VjaCBnYXBzIGFuZCB0aGVpciBlZmZv cnRzIGp1c3QgaWdub3JlZCBieSB0aGUgZGV2ZWxvcGVycy4KPj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+ Pj4gRXZlcnkgeWVhciBGcmVlQlNEIENvbW11bml0eSBTdXJ2ZXkgYXNraW5nIGFib3V0IHVzZXIg ZXhwZXJpZW5jZSBpbiBjb250cmlidXRpbmcgdG8gRnJlZUJTRC4KPj4+Pj4+Pj4+Pj4KPj4+Pj4+ Pj4+Pj4gSGVyZSB5b3UgY2FuIHNlZSBhbiBleGFtcGxlIG9mIHN1Y2ggY29udHJpYnV0aW5nLgo+ Pj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pj4gRmlyc3QsIHRoYW5rIHlvdSBmb3IgYmVpbmcgcGVyc2lzdGVu dCBhbmQgY29udGludWluZyB0byBicmluZyBpdCB1cC4gSXQncyBpbXBvcnRhbnQgdG8gZG8gdGhh dCB0byBtYWtlIHN1cmUgdGhpcyAoYW5kIHlvdXIgbWFueSBvdGhlcikgY29udHJpYnV0aW9uIGRv ZXNuJ3QgZmFsbCBvbiB0aGUgZmxvb3IuCj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+PiBBbmQgdG8gYmUg ZmFpciwgd2UncmUgb25seSAzIG1vbnRocyBzaW5jZSB0aGUgbGFzdCB1cGRhdGUuIFN0aWxsLCBx dWl0ZSBhIGJpdCBsb25nZXIgdGhhbiB5b3Ugc2hvdWxkIGhhdmUgdG8gd2FpdCwgYnV0IG5vdCBu ZWFybHkgdGhlIHllYXIgdGhlIG9yaWdpbmFsIGRhdGUgc3VnZ2VzdHMuCj4+Pj4+Pj4+Pj4KPj4+ Pj4+Pj4+PiBBbmQgdGhpcyBpcyBhIHBlcmZlY3Qgc3Rvcm0gb2YgImhvdyB0aGUgcHJvamVjdCBp cyBiYWQgYXQgYWNjZXB0aW5nIGNvbnRyaWJ1dGlvbnMiOgo+Pj4+Pj4+Pj4+ICgxKSBUaGUgb3Jp Z2luYWwgc3VibWlzc2lvbiB3YXMgY2xvc2UgdG8gdGhlIDE0IGJyYW5jaCBjcmVhdGlvbiB0aW1l LiBUaGlzIG1lYW50IHRoYXQgd2Ugd2VyZW4ndCB3ZWxsIHByZXBhcmVkIHRvIGxvb2sgYXQgaXQg c2luY2UgaXQgaXMgc3VjaCBhbiBpbnZhc2l2ZSBjaGFuZ2UgKGF0IGxlYXN0IG9uIGl0cyBzdXJm YWNlKS4gSXQgYWxzbyBzbG93ZWQgdGhlIGluaXRpYWwgcmVzcG9uc2UuLi4KPj4+Pj4+Pj4+PiAo MikgVGhlcmUgd2FzIGEgbnVtYmVyIG9mIGJhY2sgYW5kIGZvcnRoIHJlcXVlc3RzIGZvciBjaGFu Z2VzLCB3aGljaCB0b29rIHRpbWUgdG8gc29ydCBvdXQuLi4KPj4+Pj4+Pj4+PiAoMykgVGhlIHNp emUgb2YgdGhpcyBpcyBodWdlLCB3ZWxsIGJleW9uZCB0aGUgY2FwYWNpdHkgb2YgUGhhYnJpY2F0 b3IgdG8gcmV2aWV3IGFjY3VyYXRlbHkuLi4KPj4+Pj4+Pj4+PiAoNCkgSXQncyBhIHZlbmRvciBp bXBvcnQuIFRoYXQgbWVhbnMgd2UgY2FuJ3QganVzdCBkcm9wIHRoZSBQaGFicmljYXRvciByZXZp ZXcgaW50byB0aGUgdHJlZS4uLgo+Pj4+Pj4+Pj4+ICg1KSBJdCdzIHBoYWJyaWNhdG9yOiB0aGlz IGlzIGEgZ3JlYXQgdG9vbCBmb3IgZGV2ZWxvcGVycywgYnV0IHdlIGhhdmUgYSB0ZXJyaWJsZSB0 cmFjayByZWNvcmQgb2YgdXNpbmcgaXQgZm9yIGludGFrZSBmcm9tIG5ldyBjb250cmlidXRvcnMu IFdlIGRvbid0IGhhdmUgYW55IG92ZXJzaWdodCBhdCBhbGwgb3ZlciB0aGlzIHRvb2wsIGF0IHRo ZXJlJ3MgYXQgYmVzdCB0ZXBpZCBhbmQgbHVrZSB3YXJtIGF0dGVtcHRzIHRvIGxvb2sgZm9yIGRy b3AgYmFsbHMuCj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+PiBBbGwgb2YgdGhlc2UgdGhpbmdzIGFyZSBh IHRlcnJpYmxlIGV4cGVyaWVuY2UuIEkgY2FuIG9ubHkgYXBvbG9naXplLiBUaGVzZSBkYXlzLCB3 ZSBtaWdodCBzdGVlciB0aGlzIHRvd2FyZHMgZ2l0aHViLCBidXQgdGhlICd2ZW5kb3IgaW1wb3J0 JyBtZWFucyB5b3UgcmVhbGx5IG5lZWQgc29tZW9uZSBvbiB0aGUgaW5zaWRlLCBvciB5b3UgbmVl ZCB0byBiZSBvbiB0aGUgaW5zaWRlIHRvIG1ha2UgdGhhdCB3b3JrLgo+Pj4+Pj4+Pj4+Cj4+Pj4+ Pj4+Pj4gU28sIGhvdyB0byBtb3ZlIGZvcndhcmQ/IFdlbGwsIEknZCBsaWtlIHRvIHByb3Bvc2Ug dGhlIGZvbGxvd2luZzoKPj4+Pj4+Pj4+PiAoMSkgc3VibWl0IGFsbCB0aGUgb3RoZXIgUGhhYnJp Y2F0b3IgcmV2aWV3cyB5b3UgaGF2ZSBvcGVuICh0aGV5IGFyZSBtb3N0bHkgZ29vZCwgb3IgY2xv c2UgdG8gZ29vZCkgdG8gZ2l0aHViLiBHaXRodWIgaXMgYmVpbmcgYWN0aXZlbHkgbWFuYWdlZCBh bmQgd2lsbCBtYWtlIGl0IGZhc3RlciB0byBnZXQgdGhpbmdzIGl0LiBJdCdzIGEgbXVjaCBiZXR0 ZXIgdG9vbCBmb3IgbmV3IGNvbnRyaWJ1dG9ycyAoYW5kIGV2ZW4gZnJlcXVlbnQgY29udHJpYnV0 b3JzIG9mIHNtYWxsaXNoIHRoaW5ncykuCj4+Pj4+Pj4+Pj4gKDIpIEkgc2hvdWxkIGRvIGFuIHZl bmRvciBpbXBvcnQgb2YgNS4zLjAgZnJvbSBnaXRodWIsIGFuZCBkbyB0aGUgbWVyZ2UgdG8gYSBi cmFuY2ggYW5kIHB1c2ggdGhhdCB0byBnaXRodWIuIFlvdSBjYW4gdGhlbiBsYXllciBvbiB5b3Vy IGNoYW5nZXMgYW5kIHRob3NlIGNhbiBiZSByZXZpZXdlZCBtb3JlIGNsb3NlbHkgYXMgYSBwdWxs IHJlcXVlc3QgYWdhaW5zdCB0aGUgYnJhbmNoIEkgcHVzaC4gSSBzdXNwZWN0IHRoYXQgbW9zdCBv ZiB0aGUgaXNzdWVzIGFyZSBzb3J0ZWQgb3V0IGFscmVhZHkKPj4+Pj4+Pj4+PiAoMykgSSdsbCBs YW5kIGl0IHZpYSB0aGF0IHJvdXRlLi4uCj4+Pj4+Pj4+Pj4KPj4+Pj4+Pj4+PiBBbmQsIGlmIHRo ZSBzdW0gb2YgdGhlIG90aGVyIHB1bGwgcmVxdWVzdHMgYW5kIHRoaXMgYXJlIGdvb2QgKGFuZCBJ IHN1c3BlY3QgdGhleSB3aWxsIGJlKSwgdGhlbiB3ZSBjYW4gdGFsayBhYm91dCBjb21taXQgYml0 cyBhbmQgc3VjaC4KPj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4+IEl0J3MgZXhwZXJpZW5jZXMgbGlrZSB0 aGlzIHdoaWNoIGlzIHdoeSBJJ20gdHJ5aW5nIHRvIHN0YW5kIHVwIGdpdGh1YiBwdWxsIHJlcXVl c3RzIGFzIGEgcmVsaWFibGUgd2F5IHRvIGdldCB0aGluZ3MgYW5kIGFuZCB0aGUgYmVzdCBwbGFj ZSB0byBzZW5kIHBlb3BsZS4uLgo+Pj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pj4gVGhhbmtzIGFnYWluIGZv ciBwZXJzaXN0aW5nLCBhbmQgYWxzbyBmb3IgZXhwcmVzc2luZyB0aGlzIGNyaXRpY2lzbSB0aGF0 IHdlIChob3BlZnVsbHkpIGNhbiB1c2UgdG8gbWFrZSBpdCBiZXR0ZXIuCj4+Pj4+Pj4+Pj4KPj4+ Pj4+Pj4+PiBXYXJuZXIKPj4+Pj4+Pj4+Cj4+Pj4+Pj4+PiBIZWxsby4KPj4+Pj4+Pj4+Cj4+Pj4+ Pj4+PiBJJ20gbm90IHRoZSBhdXRob3Igb2YgRDQxNDIxLiBKdXN0IGFwcGxpZWQgdGhlIHBhdGNo IHRvIHRlc3QgaXQgOCBtb250aHMgYWdvLiBBbmQgcmVjZW50bHkgZGlzY292ZXJlZCB0aGF0IGl0 J3Mgc3RpbGwgbm90IGNvbW1pdHRlZC4KPj4+Pj4+Pj4+IEkgY2FuJ3QgY29weSB5b3VyIG1lc3Nh Z2UgdG8gUGhhYnJpY2F0b3IgYmVjYXVzZSBkb24ndCBoYXZlIGFuIGFjY291bnQuIFBsZWFzZSwg aWYgeW91IGhhdmUgdGltZSwgaGVscCB0aGUgYXV0aG9yIGluIEQ0MTQyMS4KPj4+Pj4+Pj4KPj4+ Pj4+Pj4gQWggeWVzLiBJJ3ZlIGJlZW4gaW4gdG91Y2ggd2l0aCB0aGUgYXV0aG9yIGZvciBvdGhl ciB0aGluZ3MsIGFuZCBzb21laG93IHRob3VnaHQgaXQgd2FzIHlvdS4uLi4gSSdsbCByZWFjaCBv dXQgdG8gaGltIHZpYSBvdGhlciBtZWFucy4uLgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiBXYXJuZXI= --b1=_LL69nXigG2smgSsejGdXqg0i84csaQ4J6T0lWPRUQ Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGRpdj5JIHJlc29sdmVkIG1lcmdlIGNvbmZsaWN0IGJ5IHJlYmFzaW5nIG1haW4uIFdoYXQncyBu ZXh0PzwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9u dC1zaXplOiAxNHB4OyBjb2xvcjogcmdiKDAsIDAsIDApOyBiYWNrZ3JvdW5kLWNvbG9yOiByZ2Io MjU1LCAyNTUsIDI1NSk7Ij48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFs LCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHg7IGNvbG9yOiByZ2IoMCwgMCwgMCk7IGJhY2tn cm91bmQtY29sb3I6IHJnYigyNTUsIDI1NSwgMjU1KTsiPjxzcGFuPjxhIHRhcmdldD0iX2JsYW5r IiByZWw9Im5vcmVmZXJyZXIgbm9mb2xsb3cgbm9vcGVuZXIiIGhyZWY9Imh0dHBzOi8vZ2l0aHVi LmNvbS9mcmVlYnNkL2ZyZWVic2Qtc3JjL3B1bGwvMTMzNyI+aHR0cHM6Ly9naXRodWIuY29tL2Zy ZWVic2QvZnJlZWJzZC1zcmMvcHVsbC8xMzM3PC9hPjwvc3Bhbj48YnI+PC9kaXY+PGRpdiBjbGFz cz0icHJvdG9ubWFpbF9xdW90ZSI+DQogICAgICAgIE9uIFN1bmRheSwgRGVjZW1iZXIgMXN0LCAy MDI0IGF0IDI6MDUgQU0sIFdhcm5lciBMb3NoICZsdDtpbXBAYnNkaW1wLmNvbSZndDsgd3JvdGU6 PGJyPg0KICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0icHJvdG9ubWFpbF9xdW90ZSIgdHlwZT0i Y2l0ZSI+DQogICAgICAgICAgICA8ZGl2IGRpcj0ibHRyIj48ZGl2IGRpcj0ibHRyIj4oc29ycnkg dG8gZm9sbG93IHVwIHRvIG15IG93biBlbWFpbCBhbmQgdG9wcG9zdGluZyk8L2Rpdj48ZGl2IGRp cj0ibHRyIj48YnI+PC9kaXY+PGRpdj5JIGdvdCB0aGUgdmVuZG9yIGJyYW5jaCBib290c3RyYXBw ZWQ6IEkgY3JlYXRlZCB2ZW5kb3IvamVtYWxsb2MgYW5kIHRhZ2dlZCB2ZW5kb3IvamVtYWxsb2Mv NS4yLjEsPC9kaXY+PGRpdj5hbmQgY3JlYXRlZCBhIG1lcmdlIGNvbW1pdCBmcm9tIHRoYXQgYnJh bmNoIHRvIG1haW4uIEkgaGFkIHRvIHR3ZWFrIHRoZTwvZGl2PjxkaXY+RlJFRUJTRC1YbGlzdCBh IGxpdHRsZS5JJ3ZlIG5vdCB1cGRhdGVkIHRoZSBvdGhlciB0d28gRlJFRUJTRC0qIGZpbGVzLCBi dXQgdGhlIHN0ZXBzIHdlcmU8L2Rpdj48ZGl2PmRvY3VtZW50ZWQgaW4gdGhlIGNvbW1pdCBtZXNz YWdlcyAodGhlIHZlbmRvciBicmFuY2ggb25lIGlzIGVzcGVjaWFsbHkgbG9uZyBhbmQ8L2Rpdj48 ZGl2Pmxpa2VseSBzaG91bGQgbWlncmF0ZSBpbnRvIHRoZSBkZXZlbG9wZXIgaGFuZGJvb2spLiBG UkVFQlNELXVwZGF0ZSBpcyBiYXNpY2FsbHk8L2Rpdj48ZGl2PmEgc2hlbGwgc2NyaXB0IHRvIGRv IHRoZSBzYW1lIHRoaW5nIHRoYXQgZ2l0IHN1YnRyZWUgbWVyZ2UgZG9lcywgdGhvdWdoIEknbSBz dXJlIHNvbWU8L2Rpdj48ZGl2PnR3ZWFrcyBjb3VsZCBoZWxwIHRoZSBuZXh0IHRpbWUgKGlmIHRo ZXJlIGlzIGEgbmV4dCB0aW1lLCBqZW1hbGxvYyB1cHN0cmVhbSBzZWVtcyB0bzwvZGl2PjxkaXY+ aGF2ZSBzbG93ZWQgd2F5IGRvd24gb2YgbGF0ZSkuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5O ZXh0IHVwLEknbGwgY3JlYXRlIDUuMy4wIGltcG9ydCBvbiB0aGUgdmVuZG9yIGJyYW5jaCwgZG8g dGhlIG1lcmdlIGFuZCBzdGFydCB0ZXN0aW5nIChpdCB3aWxsIGJlPC9kaXY+PGRpdj5NaW5zb28n cyBwdWxsIHJlcXVlc3QsIHJlYmFzZWQsIHdpdGggYW55IGNvbmZsaWN0cyByZXNvbHZlZCBhbmQg bWVyZ2UgY29tbWl0IHJlY29yZGVkKS48L2Rpdj48ZGl2PkJ1dCB0aGF0IHdpbGwgaGF2ZSB0byBi ZSB0b21vcnJvdyBvciBtb3JlIGxpa2VseSBkdXJpbmcgdGhlIHdvcmsgd2Vlay4gSSdtIHRvbyB0 aXJlZCB0b25pZ2h0PC9kaXY+PGRpdj50byBnZXQgaXQgcmlnaHQgYXQgdGhlIG1vbWVudC48L2Rp dj48ZGl2Pjxicj48L2Rpdj48ZGl2PkFuZCBhIHNwZWNpYWwgdGhhbmtzIHRvIGVtYXN0ZSBmb3Ig Z2l2aW5nIGJ6IHRoZSByaWdodCByZWNpcGUgZm9yIGRvaW5nIHRoZSBzdWJ0cmVlIG1lcmdlPC9k aXY+PGRpdj53L28gZ2l0IHN1YnRyZWUgZm9yIGhpcyB3b3JrIG9uIHRoZSBsaW51eCB3aWZpIGRy aXZlcnMgaW4gdGhlIHRyZWUuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5XYXJuZXI8L2Rpdj48 YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUgZ21haWxfcXVvdGVfY29udGFpbmVyIj48ZGl2IGNs YXNzPSJnbWFpbF9hdHRyIiBkaXI9Imx0ciI+T24gU2F0LCBOb3YgMzAsIDIwMjQgYXQgOTozOOKA r1BNIFdhcm5lciBMb3NoICZsdDs8YSBocmVmPSJtYWlsdG86aW1wQGJzZGltcC5jb20iIHJlbD0i bm9yZWZlcnJlciBub2ZvbGxvdyBub29wZW5lciI+aW1wQGJzZGltcC5jb208L2E+Jmd0OyB3cm90 ZTo8YnI+PC9kaXY+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDti b3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4IiBj bGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIiPlllYSwgSSBuZWVkIHRvIGdldCBhIGNv cHkgb2YgamVtYWxsb2MgNS4zLjAgYW5kIDUuMi4xIHRvIHRyeSB0byAnYm9vdHN0cmFwJyB0aGUg dmVuZG9yIGJyYW5jaC48ZGl2PlRoZW4gSSBuZWVkIHRvIGJvb3RzdHJhcCBpdC4uLjwvZGl2Pjxk aXY+PGJyPjwvZGl2PjxkaXY+SSBqdXN0IGRpZCB0aGUgc2FtZSB3aXRoIGVkazIgKHdoaWNoIGhh ZCBhIHZlbmRvciBicmFuY2gsIGJ1dCBoYWRuJ3QgYmVlbiB1cGRhdGVkIHNpbmNlIHN2biB0aW1l cykuPC9kaXY+PGRpdj5Ib3dldmVyLCBqZW1hbGxvYyBkb2Vzbid0IGhhdmUgYSB2ZW5kb3IgYnJh bmNoIHlldCwgc28gSSdsbCBoYXZlIHRvIGNyZWF0ZSB0aGF0LCBidXQgSSdsbCBzdGFydCB3aXRo IHRoZTwvZGl2PjxkaXY+Y3VycmVudCB2ZXJzaW9uIHJhdGhlciB0aGFuIGRvaW5nIGZ1bGwgaGlz dG9yeS4uLiAgU28gSSdsbCBzdGFydCB0aGVyZS48L2Rpdj48ZGl2PkkgYWxzbyBqdXN0IGRpZCBh d2sgYW5kIGx1YSwgc28gb25jZSBJIGhhdmUgdGhpbmdzIGJvb3RzdHJhcHBlZCwgSSdsbCBiZSBh YmxlIHRvIGFkZCA1LjMuMCBhbmQgdGhlbiBsYXllciBNaW5zb28ncyAgb248L2Rpdj48ZGl2PnRv cCBvZiB0aGF0IGFuZCB0aGVuIHN0YXJ0IHRlc3RpbmcgaXQgc29tZWhvdy48L2Rpdj48ZGl2Pjxi cj48L2Rpdj48ZGl2Pk1hbGxvYyBtYWtlcyBtZSBuZXJ2b3VzIHRvIHRvdWNoLCBob25lc3RseSwg YnV0IEknbGwgZ2l2ZSBpdCBhIGdvIGFuZCB0ZXN0IGJvb3Qgb24gbXkgc3lzdGVtIGFuZDwvZGl2 PjxkaXY+bWF5YmUgc2VlIGlmIHdlIGNhbiBzdXJ2aXZlIGEgd29ya2xvYWQgYXQgd29yayB3L28g cmVncmVzc2lvbnMuLi4gQnV0IEkgY2FuJ3QgZG8gYSBmdWxsIHRlc3Qgd2l0aCBsb3RzPGJyPjwv ZGl2PjxkaXY+b2YgbWFjaGluZXMgdW50aWwgYWZ0ZXIgdGhlIGZpcnN0IG9mIHRoZSB5ZWFyICh0 aG91Z2ggSSBjYW4gZG8gYSBjb3VwbGUgZm9yIGEgZmV3IGRheXMgYmVmb3JlIHRoZW4pLjwvZGl2 PjxkaXY+PGJyPjwvZGl2PjxkaXY+U28gbXkgbmV4dCBzdGVwIGlzIHRvIGJvb3RzdHJhcCB0aGUg dmVuZG9yIGJyYW5jaC4uLiBJJ2xsIGdpdmUgdGhhdCBhIHRyeSB0b25pZ2h0LjwvZGl2PjxkaXY+ PGJyPjwvZGl2PjxkaXY+V2FybmVyPC9kaXY+PC9kaXY+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1 b3RlIj48ZGl2IGNsYXNzPSJnbWFpbF9hdHRyIiBkaXI9Imx0ciI+T24gU2F0LCBOb3YgMzAsIDIw MjQgYXQgODoyNuKAr1BNIE1pbnNvbyBDaG9vICZsdDs8YSB0YXJnZXQ9Il9ibGFuayIgaHJlZj0i bWFpbHRvOm1pbnNvb2Nob28wMTIyQHByb3Rvbi5tZSIgcmVsPSJub3JlZmVycmVyIG5vZm9sbG93 IG5vb3BlbmVyIj5taW5zb29jaG9vMDEyMkBwcm90b24ubWU8L2E+Jmd0OyB3cm90ZTo8YnI+PC9k aXY+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVm dDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4IiBjbGFzcz0iZ21h aWxfcXVvdGUiPjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7Zm9udC1z aXplOjE0cHg7Y29sb3I6cmdiKDAsMCwwKTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1 NSkiPkkgaGF2ZSBhbHJlYWR5IHN1Ym1pdHRlZCBQUiBvbiBnaXRodWIgKDxzcGFuPjxhIHRhcmdl dD0iX2JsYW5rIiBocmVmPSJodHRwczovL2dpdGh1Yi5jb20vZnJlZWJzZC9mcmVlYnNkLXNyYy9w dWxsLzEzMzciIHJlbD0ibm9yZWZlcnJlciBub2ZvbGxvdyBub29wZW5lciI+aHR0cHM6Ly9naXRo dWIuY29tL2ZyZWVic2QvZnJlZWJzZC1zcmMvcHVsbC8xMzM3PC9hPjwvc3Bhbj4pIGFuZCBwaGFi cmljYXRvciAoPHNwYW4+PGEgdGFyZ2V0PSJfYmxhbmsiIGhyZWY9Imh0dHBzOi8vcmV2aWV3cy5m cmVlYnNkLm9yZy9ENDE0MjEiIHJlbD0ibm9yZWZlcnJlciBub2ZvbGxvdyBub29wZW5lciI+aHR0 cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0Q0MTQyMTwvYT48L3NwYW4+KS4gSSBkb24ndCBoYXZl IGFjY2VzcyAoY29tbWl0IGJpdCkgdG8gZnJlZWJzZCBnaXQgcmVwbywgc28gdGhlcmUgaXMgbm90 aGluZyBJIGNhbiBkbyBhdCB0aGlzIHBvaW50IHNpbmNlIHZlbmRvciBpbXBvcnQgYW5kIGxhbmRp bmcgcGF0Y2hlcyByZXF1aXJlcyBjb21taXQgYml0Ljxicj48L2Rpdj48ZGl2Pg0KICAgICAgICBP biBTYXR1cmRheSwgTm92ZW1iZXIgMzB0aCwgMjAyNCBhdCAxOjQyIFBNLCBjZ2xvZ2ljICZsdDs8 YSB0YXJnZXQ9Il9ibGFuayIgaHJlZj0ibWFpbHRvOmNnbG9naWNAcHJvdG9ubWFpbC5jb20iIHJl bD0ibm9yZWZlcnJlciBub2ZvbGxvdyBub29wZW5lciI+Y2dsb2dpY0Bwcm90b25tYWlsLmNvbTwv YT4mZ3Q7IHdyb3RlOjxicj4NCiAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+DQogICAg ICAgICAgICA8ZGl2IHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmO2ZvbnQtc2l6 ZToxNHB4Ij5JIHNlZSwgaXQgaGFwcGVucy48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTpB cmlhbCxzYW5zLXNlcmlmO2ZvbnQtc2l6ZToxNHB4Ij48c3Bhbj5NYXliZSBhbm90aGVyIGNvbW1p dHRlciB3aWxsIHZvbHVudGVlciB0byBkbyB0aGUgdXBkYXRlLjwvc3Bhbj48ZGl2PkkgaG9wZSBp dCB3aWxsIG1ha2UgaXRzIHdheSBpbnRvIDE1LjAgcmVsZWFzZS48L2Rpdj48ZGl2Pjxicj48L2Rp dj48ZGl2PlRoYW5rcy48L2Rpdj48L2Rpdj48ZGl2Pg0KICAgICAgICBPbiBGcmlkYXksIE5vdmVt YmVyIDI5dGgsIDIwMjQgYXQgOTozOCBQTSwgV2FybmVyIExvc2ggJmx0OzxhIHRhcmdldD0iX2Js YW5rIiBocmVmPSJtYWlsdG86aW1wQGJzZGltcC5jb20iIHJlbD0ibm9yZWZlcnJlciBub2ZvbGxv dyBub29wZW5lciI+aW1wQGJzZGltcC5jb208L2E+Jmd0OyB3cm90ZTo8YnI+DQogICAgICAgIDxi bG9ja3F1b3RlIHR5cGU9ImNpdGUiPg0KICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+SSd2ZSBi ZWVuIHN3YW1wZWQuIHdlIG5lZWQgdG8gYm9vdHN0cmFwIHRoZSB2ZW5kb3IgYnJhbmNoLCBhbmQg dGhlIHdheSBwcmlvciB1cGRhdGVzIHdlcmUgZG9uZTxkaXY+aXNuJ3Qgc28gZ3JlYXQuIDwvZGl2 PjxkaXY+PGJyPjwvZGl2PjxkaXY+V2FybmVyPC9kaXY+PC9kaXY+PGJyPjxkaXYgY2xhc3M9Imdt YWlsX3F1b3RlIj48ZGl2IGNsYXNzPSJnbWFpbF9hdHRyIiBkaXI9Imx0ciI+T24gTW9uLCBOb3Yg MjUsIDIwMjQgYXQgMjoyMeKAr0FNIGNnbG9naWMgJmx0OzxhIHRhcmdldD0iX2JsYW5rIiBocmVm PSJtYWlsdG86Y2dsb2dpY0Bwcm90b25tYWlsLmNvbSIgcmVsPSJub3JlZmVycmVyIG5vZm9sbG93 IG5vb3BlbmVyIj5jZ2xvZ2ljQHByb3Rvbm1haWwuY29tPC9hPiZndDsgd3JvdGU6PGJyPjwvZGl2 PjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6 MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9ImdtYWls X3F1b3RlIj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmO2ZvbnQtc2l6 ZToxNHB4Ij5IZWxsbyBndXlzLDwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNh bnMtc2VyaWY7Zm9udC1zaXplOjE0cHgiPjxicj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWls eTpBcmlhbCxzYW5zLXNlcmlmO2ZvbnQtc2l6ZToxNHB4Ij5Ib3cgdGhlIHVwZGF0ZSBvZiBqZW1h bGxvYyBpcyBnb2luZz8gSXQncyBOb3ZlbWJlciBub3cuPC9kaXY+PGRpdiBzdHlsZT0iZm9udC1m YW1pbHk6QXJpYWwsc2Fucy1zZXJpZjtmb250LXNpemU6MTRweCI+PGJyPjwvZGl2PjxkaXYgc3R5 bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7Zm9udC1zaXplOjE0cHgiPlRoYW5rcy48 L2Rpdj48ZGl2Pg0KICAgICAgICBPbiBNb25kYXksIEp1bHkgMjJuZCwgMjAyNCBhdCA3OjAyIFBN LCBNaW5zb28gQ2hvbyAmbHQ7PGEgdGFyZ2V0PSJfYmxhbmsiIGhyZWY9Im1haWx0bzptaW5zb29j aG9vMDEyMkBwcm90b24ubWUiIHJlbD0ibm9yZWZlcnJlciBub2ZvbGxvdyBub29wZW5lciI+bWlu c29vY2hvbzAxMjJAcHJvdG9uLm1lPC9hPiZndDsgd3JvdGU6PGJyPg0KICAgICAgICA8YmxvY2tx dW90ZSB0eXBlPSJjaXRlIj4NCiAgICAgICAgICAgIDxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFy aWFsLHNhbnMtc2VyaWY7Zm9udC1zaXplOjE0cHgiPkZpcnN0LCBzb3JyeSBmb3IgbGF0ZSByZXNw b25zZS48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmO2ZvbnQt c2l6ZToxNHB4Ij48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1z ZXJpZjtmb250LXNpemU6MTRweCI+Y2dsb2dpYywgdGhhbmsgeW91IGZvciBicmluZ2luZyB1cCB0 aGlzIGlzc3VlIGFnYWluIHNpbmNlIEkgbmVhcmx5IGZvcmdvdCB0aGF0IHRoaXMgaXNzdWUgd2Fz IHN0aWxsIG9wZW4uPC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJp Zjtmb250LXNpemU6MTRweCI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFyaWFs LHNhbnMtc2VyaWY7Zm9udC1zaXplOjE0cHgiPldhcm5lciwgYXMgSSBjYW4ndCBhY2Nlc3MgdG8g bXkgRnJlZUJTRCBpbnN0YW5jZSB1bnRpbCB0aGUgZW5kIG9mIEF1Z3VzdCwgYnV0IEkgY2FuIHN0 aWxsIGVkaXQgYW5kIHB1c2ggdGhlIGNvZGUgdGhyb3VnaCBteSBBcm0gTWFjLiBUaGlzIG1lYW5z IHRoYXQgSSBjYW4ndCB0ZXN0IHRoZSB1cGRhdGVkIGNvZGUgb24gbXkgbWFjaGluZSwgYnV0IEkg Y2FuIGpvaW4gdGhlIHJldmlldyBwcm9jZXNzIGFuZCBsaXN0ZW4gdG8gY2hhbmdlIHByb3Bvc2Fs cy48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmO2ZvbnQtc2l6 ZToxNHB4Ij48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJp Zjtmb250LXNpemU6MTRweCI+SSdsbCBvcGVuIGEgR2l0aHViIFBSIGluIGEgZmV3IGhvdXJzLiAo VGhlIHBoYWJyaWNhdG9yIHJldmlldyB3aWxsIHN0YXkgb3BlbmVkIGp1c3QgaW4gY2FzZSk8L2Rp dj48ZGl2Pg0KICAgICAgICBPbiBNb25kYXksIEp1bHkgMjJuZCwgMjAyNCBhdCA1OjA4IEFNLCBX YXJuZXIgTG9zaCAmbHQ7PGEgdGFyZ2V0PSJfYmxhbmsiIGhyZWY9Im1haWx0bzppbXBAYnNkaW1w LmNvbSIgcmVsPSJub3JlZmVycmVyIG5vZm9sbG93IG5vb3BlbmVyIj5pbXBAYnNkaW1wLmNvbTwv YT4mZ3Q7IHdyb3RlOjxicj4NCiAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+DQogICAg ICAgICAgICA8ZGl2IGRpcj0ibHRyIj48ZGl2IGRpcj0ibHRyIj48YnI+PC9kaXY+PGJyPjxkaXYg Y2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IGNsYXNzPSJnbWFpbF9hdHRyIiBkaXI9Imx0ciI+T24g U3VuLCBKdWwgMjEsIDIwMjQgYXQgMjowM+KAr1BNIGNnbG9naWMgJmx0OzxhIHRhcmdldD0iX2Js YW5rIiBocmVmPSJtYWlsdG86Y2dsb2dpY0Bwcm90b25tYWlsLmNvbSIgcmVsPSJub3JlZmVycmVy IG5vZm9sbG93IG5vb3BlbmVyIj5jZ2xvZ2ljQHByb3Rvbm1haWwuY29tPC9hPiZndDsgd3JvdGU6 PGJyPjwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9y ZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xh c3M9ImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlm O2ZvbnQtc2l6ZToxNHB4Ij48YnI+PC9kaXY+PGRpdj4NCiAgICAgICAgT24gU3VuZGF5LCBKdWx5 IDIxc3QsIDIwMjQgYXQgNjo1NCBBTSwgV2FybmVyIExvc2ggJmx0OzxhIHRhcmdldD0iX2JsYW5r IiBocmVmPSJtYWlsdG86aW1wQGJzZGltcC5jb20iIHJlbD0ibm9yZWZlcnJlciBub2ZvbGxvdyBu b29wZW5lciI+aW1wQGJzZGltcC5jb208L2E+Jmd0OyB3cm90ZTo8YnI+DQogICAgICAgIDxibG9j a3F1b3RlIHR5cGU9ImNpdGUiPg0KICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9 Imx0ciI+PGJyPjwvZGl2Pjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+PGRpdiBjbGFzcz0i Z21haWxfYXR0ciIgZGlyPSJsdHIiPk9uIFNhdCwgSnVsIDIwLCAyMDI0IGF0IDE6NTnigK9BTSBj Z2xvZ2ljICZsdDs8YSB0YXJnZXQ9Il9ibGFuayIgaHJlZj0ibWFpbHRvOmNnbG9naWNAcHJvdG9u bWFpbC5jb20iIHJlbD0ibm9yZWZlcnJlciBub2ZvbGxvdyBub29wZW5lciI+Y2dsb2dpY0Bwcm90 b25tYWlsLmNvbTwvYT4mZ3Q7IHdyb3RlOjxicj48L2Rpdj48YmxvY2txdW90ZSBzdHlsZT0ibWFy Z2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwy MDQpO3BhZGRpbmctbGVmdDoxZXgiIGNsYXNzPSJnbWFpbF9xdW90ZSI+PGRpdiBzdHlsZT0iZm9u dC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjtmb250LXNpemU6MTRweDtjb2xvcjpyZ2IoMCwwLDAp O2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+SGVsbG8gRnJlZUJTRCBjb21tdW5p dHksPC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjtmb250LXNp emU6MTRweDtjb2xvcjpyZ2IoMCwwLDApO2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1 KSI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7Zm9u dC1zaXplOjE0cHg7Y29sb3I6cmdiKDAsMCwwKTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1 LDI1NSkiPjxzcGFuIHN0eWxlPSJkaXNwbGF5OmlubGluZTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigy NTUsMjU1LDI1NSkiPkFmdGVyIDwvc3Bhbj48c3BhbiBzdHlsZT0iYmFja2dyb3VuZC1jb2xvcjpy Z2IoMjU1LDI1NSwyNTUpIj5KYXNvbiBFdmFucyBzdGVwcGVkIGFzaWRlIGZyb20gbWFpbnRhaW5p bmcgamVtYWxsb2MgaW4gRnJlZUJTRCwgaXQncyBub3QgdXBkYXRpbmcgaW4gdGltZSBhbnltb3Jl Ljwvc3Bhbj48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJp Zjtmb250LXNpemU6MTRweDtjb2xvcjpyZ2IoMCwwLDApO2JhY2tncm91bmQtY29sb3I6cmdiKDI1 NSwyNTUsMjU1KSI+VmVyc2lvbiA1LjMuMCB3YXMgcmVsZWFzZWQgPHNwYW4+TWF5IDYsIDIwMjIg YW5kIEZyZWVCU0Qgc3RpbGwgbm90IGltcG9ydGVkIGl0IGludG8gdGhlIHRyZWUuPC9zcGFuPjwv ZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7Zm9udC1zaXplOjE0 cHg7Y29sb3I6cmdiKDAsMCwwKTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1NSkiPjxz cGFuPjxicj48L3NwYW4+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1z ZXJpZjtmb250LXNpemU6MTRweDtjb2xvcjpyZ2IoMCwwLDApO2JhY2tncm91bmQtY29sb3I6cmdi KDI1NSwyNTUsMjU1KSI+VGhlcmUgaXMgYSBwZW5kaW5nIHJldmlldyA8c3Bhbj48YSB0YXJnZXQ9 Il9ibGFuayIgaHJlZj0iaHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0Q0MTQyMSIgcmVsPSJu b3JlZmVycmVyIG5vZm9sbG93IG5vb3BlbmVyIj5odHRwczovL3Jldmlld3MuZnJlZWJzZC5vcmcv RDQxNDIxPC9hPiBmcm9tIDxzcGFuPkF1ZyAxMSwgMjAyMy48L3NwYW4+PC9zcGFuPjwvZGl2Pjxk aXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7Zm9udC1zaXplOjE0cHg7Y29s b3I6cmdiKDAsMCwwKTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1NSkiPjxzcGFuPjxz cGFuPkknbSBzdWNjZXNzZnVsbHkgcnVubmluZyBGcmVlQlNEL2FtZDY0IHN5c3RlbSB3aXRoIEQ0 MTQyMSBhcHBsaWVkIGZvciA4IG1vbnRocywgYXMgd2VsbCBhcyBtYW55IG90aGVyIHBlb3BsZS48 L3NwYW4+PC9zcGFuPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2Vy aWY7Zm9udC1zaXplOjE0cHg7Y29sb3I6cmdiKDAsMCwwKTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigy NTUsMjU1LDI1NSkiPjxzcGFuPjxzcGFuPjxicj48L3NwYW4+PC9zcGFuPjwvZGl2PjxkaXYgc3R5 bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7Zm9udC1zaXplOjE0cHg7Y29sb3I6cmdi KDAsMCwwKTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1NSkiPjxzcGFuPjxzcGFuPkNh biBpdCBiZSByZXZpZXdlZCBhbmQgY29tbWl0dGVkIHRvIENVUlJFTlQ/PC9zcGFuPjwvc3Bhbj48 L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmO2ZvbnQtc2l6ZTox NHB4O2NvbG9yOnJnYigwLDAsMCk7YmFja2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj48 c3Bhbj48c3Bhbj5PciwgaWYgdGhlcmUgaXMgbm8gY29tbWl0dGVycyB3aWxsaW5nIHRvIGRvIGl0 LCBjYW4gY29tbWl0IGJpdCBiZSBnaXZlbiB0byBzdWJtaXR0ZXIgb3IgYW5vdGhlciBwZXJzb24g d2lsbGluZyB0byBkbyB0aGlzPzwvc3Bhbj48L3NwYW4+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1m YW1pbHk6QXJpYWwsc2Fucy1zZXJpZjtmb250LXNpemU6MTRweDtjb2xvcjpyZ2IoMCwwLDApO2Jh Y2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+PHNwYW4+PHNwYW4+PGJyPjwvc3Bhbj48 L3NwYW4+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjtmb250 LXNpemU6MTRweDtjb2xvcjpyZ2IoMCwwLDApO2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUs MjU1KSI+PHNwYW4+PHNwYW4+PHNwYW4+SXQncyB2ZXJ5IGRpc2FwcG9pbnRpbmcgd2hlbiB1c2Vy cyBzcGVuZCB0aGVpciB0aW1lIHRvIGZpbGwgc3VjaCBnYXBzIGFuZCB0aGVpciBlZmZvcnRzIGp1 c3QgaWdub3JlZCBieSB0aGUgZGV2ZWxvcGVycy48L3NwYW4+PGJyPjwvc3Bhbj48L3NwYW4+PC9k aXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjtmb250LXNpemU6MTRw eDtjb2xvcjpyZ2IoMCwwLDApO2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+PHNw YW4+PHNwYW4+PHNwYW4+PHNwYW4+RXZlcnkgeWVhciBGcmVlQlNEIENvbW11bml0eSBTdXJ2ZXkg YXNraW5nIGFib3V0IHVzZXIgZXhwZXJpZW5jZSBpbiBjb250cmlidXRpbmcgdG8gRnJlZUJTRC4g PC9zcGFuPjxicj48L3NwYW4+PC9zcGFuPjwvc3Bhbj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZh bWlseTpBcmlhbCxzYW5zLXNlcmlmO2ZvbnQtc2l6ZToxNHB4O2NvbG9yOnJnYigwLDAsMCk7YmFj a2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj48c3Bhbj48c3Bhbj48c3Bhbj48c3Bhbj5I ZXJlIHlvdSBjYW4gc2VlIGFuIGV4YW1wbGUgb2Ygc3VjaCBjb250cmlidXRpbmcuPC9zcGFuPjwv c3Bhbj48L3NwYW4+PC9zcGFuPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNh bnMtc2VyaWY7Zm9udC1zaXplOjE0cHg7Y29sb3I6cmdiKDAsMCwwKTtiYWNrZ3JvdW5kLWNvbG9y OnJnYigyNTUsMjU1LDI1NSkiPjxzcGFuPjxzcGFuPjxzcGFuPjwvc3Bhbj48L3NwYW4+PC9zcGFu PjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7Zm9udC1zaXpl OjE0cHg7Y29sb3I6cmdiKDAsMCwwKTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1NSki PjxzcGFuPjxicj48L3NwYW4+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+PGJyPjwvZGl2PjxkaXY+ Rmlyc3QsIHRoYW5rIHlvdSBmb3IgYmVpbmcgcGVyc2lzdGVudCBhbmQgY29udGludWluZyB0byBi cmluZyBpdCB1cC4gSXQncyBpbXBvcnRhbnQgdG8gZG8gdGhhdCB0byBtYWtlIHN1cmUgdGhpcyAo YW5kIHlvdXIgbWFueSBvdGhlcikgY29udHJpYnV0aW9uIGRvZXNuJ3QgZmFsbCBvbiB0aGUgZmxv b3IuPGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+QW5kIHRvIGJlIGZhaXIsIHdlJ3JlIG9u bHkgMyBtb250aHMgc2luY2UgdGhlIGxhc3QgdXBkYXRlLiBTdGlsbCwgcXVpdGUgYSBiaXQgbG9u Z2VyIHRoYW4geW91IHNob3VsZCBoYXZlIHRvIHdhaXQsIGJ1dCBub3QgbmVhcmx5IHRoZSB5ZWFy IHRoZSBvcmlnaW5hbCBkYXRlIHN1Z2dlc3RzLjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2 PkFuZCB0aGlzIGlzIGEgcGVyZmVjdCBzdG9ybSBvZiAiaG93IHRoZSBwcm9qZWN0IGlzIGJhZCBh dCBhY2NlcHRpbmcgY29udHJpYnV0aW9ucyI6PC9kaXY+PGRpdj4oMSkgVGhlIG9yaWdpbmFsIHN1 Ym1pc3Npb24gd2FzIGNsb3NlIHRvIHRoZSAxNCBicmFuY2ggY3JlYXRpb24gdGltZS4gVGhpcyBt ZWFudCB0aGF0IHdlIHdlcmVuJ3Qgd2VsbCBwcmVwYXJlZCB0byBsb29rIGF0IGl0IHNpbmNlIGl0 IGlzIHN1Y2ggYW4gaW52YXNpdmUgY2hhbmdlIChhdCBsZWFzdCBvbiBpdHMgc3VyZmFjZSkuIEl0 IGFsc28gc2xvd2VkIHRoZSBpbml0aWFsIHJlc3BvbnNlLi4uPGJyPjwvZGl2PjxkaXY+KDIpIFRo ZXJlIHdhcyBhIG51bWJlciBvZiBiYWNrIGFuZCBmb3J0aCByZXF1ZXN0cyBmb3IgY2hhbmdlcywg d2hpY2ggdG9vayB0aW1lIHRvIHNvcnQgb3V0Li4uPC9kaXY+PGRpdj4oMykgVGhlIHNpemUgb2Yg dGhpcyBpcyBodWdlLCB3ZWxsIGJleW9uZCB0aGUgY2FwYWNpdHkgb2YgUGhhYnJpY2F0b3IgdG8g cmV2aWV3IGFjY3VyYXRlbHkuLi48L2Rpdj48ZGl2Pig0KSBJdCdzIGEgdmVuZG9yIGltcG9ydC4g VGhhdCBtZWFucyB3ZSBjYW4ndCBqdXN0IGRyb3AgdGhlIFBoYWJyaWNhdG9yIHJldmlldyBpbnRv IHRoZSB0cmVlLi4uPC9kaXY+PGRpdj4oNSkgSXQncyBwaGFicmljYXRvcjogdGhpcyBpcyBhIGdy ZWF0IHRvb2wgZm9yIGRldmVsb3BlcnMsIGJ1dCB3ZSBoYXZlIGEgdGVycmlibGUgdHJhY2sgcmVj b3JkIG9mIHVzaW5nIGl0IGZvciBpbnRha2UgZnJvbSBuZXcgY29udHJpYnV0b3JzLiBXZSBkb24n dCBoYXZlIGFueSBvdmVyc2lnaHQgYXQgYWxsIG92ZXIgdGhpcyB0b29sLCBhdCB0aGVyZSdzIGF0 IGJlc3QgdGVwaWQgYW5kIGx1a2Ugd2FybSBhdHRlbXB0cyB0byBsb29rIGZvciBkcm9wIGJhbGxz LjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+QWxsIG9mIHRoZXNlIHRoaW5ncyBhcmUgYSB0ZXJy aWJsZSBleHBlcmllbmNlLiBJIGNhbiBvbmx5IGFwb2xvZ2l6ZS4gVGhlc2UgZGF5cywgd2UgbWln aHQgc3RlZXIgdGhpcyB0b3dhcmRzIGdpdGh1YiwgYnV0IHRoZSAndmVuZG9yIGltcG9ydCcgbWVh bnMgeW91IHJlYWxseSBuZWVkIHNvbWVvbmUgb24gdGhlIGluc2lkZSwgb3IgeW91IG5lZWQgdG8g YmUgb24gdGhlIGluc2lkZSB0byBtYWtlIHRoYXQgd29yay48L2Rpdj48ZGl2Pjxicj48L2Rpdj48 ZGl2PlNvLCBob3cgdG8gbW92ZSBmb3J3YXJkPyBXZWxsLCBJJ2QgbGlrZSB0byBwcm9wb3NlIHRo ZSBmb2xsb3dpbmc6PC9kaXY+PGRpdj4oMSkgc3VibWl0IGFsbCB0aGUgb3RoZXIgUGhhYnJpY2F0 b3IgcmV2aWV3cyB5b3UgaGF2ZSBvcGVuICh0aGV5IGFyZSBtb3N0bHkgZ29vZCwgb3IgY2xvc2Ug dG8gZ29vZCkgdG8gZ2l0aHViLiBHaXRodWIgaXMgYmVpbmcgYWN0aXZlbHkgbWFuYWdlZCBhbmQg d2lsbCBtYWtlIGl0IGZhc3RlciB0byBnZXQgdGhpbmdzIGl0LiBJdCdzIGEgbXVjaCBiZXR0ZXIg dG9vbCBmb3IgbmV3IGNvbnRyaWJ1dG9ycyAoYW5kIGV2ZW4gZnJlcXVlbnQgY29udHJpYnV0b3Jz IG9mIHNtYWxsaXNoIHRoaW5ncykuPC9kaXY+PGRpdj4oMikgSSBzaG91bGQgZG8gYW4gdmVuZG9y IGltcG9ydCBvZiA1LjMuMCBmcm9tIGdpdGh1YiwgYW5kIGRvIHRoZSBtZXJnZSB0byBhIGJyYW5j aCBhbmQgcHVzaCB0aGF0IHRvIGdpdGh1Yi4gWW91IGNhbiB0aGVuIGxheWVyIG9uIHlvdXIgY2hh bmdlcyBhbmQgdGhvc2UgY2FuIGJlIHJldmlld2VkIG1vcmUgY2xvc2VseSBhcyBhIHB1bGwgcmVx dWVzdCBhZ2FpbnN0IHRoZSBicmFuY2ggSSBwdXNoLiBJIHN1c3BlY3QgdGhhdCBtb3N0IG9mIHRo ZSBpc3N1ZXMgYXJlIHNvcnRlZCBvdXQgYWxyZWFkeSA8YnI+PC9kaXY+PGRpdj4oMykgSSdsbCBs YW5kIGl0IHZpYSB0aGF0IHJvdXRlLi4uPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5BbmQsIGlm IHRoZSBzdW0gb2YgdGhlIG90aGVyIHB1bGwgcmVxdWVzdHMgYW5kIHRoaXMgYXJlIGdvb2QgKGFu ZCBJIHN1c3BlY3QgdGhleSB3aWxsIGJlKSwgdGhlbiB3ZSBjYW4gdGFsayBhYm91dCBjb21taXQg Yml0cyBhbmQgc3VjaC48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pkl0J3MgZXhwZXJpZW5jZXMg bGlrZSB0aGlzIHdoaWNoIGlzIHdoeSBJJ20gdHJ5aW5nIHRvIHN0YW5kIHVwIGdpdGh1YiBwdWxs IHJlcXVlc3RzIGFzIGEgcmVsaWFibGUgd2F5IHRvIGdldCB0aGluZ3MgYW5kIGFuZCB0aGUgYmVz dCBwbGFjZSB0byBzZW5kIHBlb3BsZS4uLiAgPGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+ VGhhbmtzIGFnYWluIGZvciBwZXJzaXN0aW5nLCBhbmQgYWxzbyBmb3IgZXhwcmVzc2luZyB0aGlz IGNyaXRpY2lzbSB0aGF0IHdlIChob3BlZnVsbHkpIGNhbiB1c2UgdG8gbWFrZSBpdCBiZXR0ZXIu PGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+V2FybmVyPGJyPjwvZGl2PjwvZGl2PjwvZGl2 Pg0KDQogICAgICAgIDwvYmxvY2txdW90ZT48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTpB cmlhbCxzYW5zLXNlcmlmO2ZvbnQtc2l6ZToxNHB4O2NvbG9yOnJnYigwLDAsMCk7YmFja2dyb3Vu ZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1p bHk6QXJpYWwsc2Fucy1zZXJpZjtmb250LXNpemU6MTRweDtjb2xvcjpyZ2IoMCwwLDApO2JhY2tn cm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+SGVsbG8uPC9kaXY+PGRpdiBzdHlsZT0iZm9u dC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjtmb250LXNpemU6MTRweDtjb2xvcjpyZ2IoMCwwLDAp O2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9 ImZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7Zm9udC1zaXplOjE0cHg7Y29sb3I6cmdiKDAs MCwwKTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1NSkiPkknbSBub3QgdGhlIGF1dGhv ciBvZiA8c3Bhbj5ENDE0MjEuIEp1c3QgYXBwbGllZCB0aGUgcGF0Y2ggdG8gdGVzdCBpdCA4IG1v bnRocyBhZ28uIEFuZCByZWNlbnRseSBkaXNjb3ZlcmVkIHRoYXQgaXQncyBzdGlsbCBub3QgY29t bWl0dGVkLjwvc3Bhbj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNl cmlmO2ZvbnQtc2l6ZToxNHB4O2NvbG9yOnJnYigwLDAsMCk7YmFja2dyb3VuZC1jb2xvcjpyZ2Io MjU1LDI1NSwyNTUpIj48c3Bhbj5JIGNhbid0IGNvcHkgeW91ciBtZXNzYWdlIHRvIFBoYWJyaWNh dG9yIGJlY2F1c2UgZG9uJ3QgaGF2ZSBhbiBhY2NvdW50LiA8L3NwYW4+UGxlYXNlLCBpZiB5b3Ug aGF2ZSB0aW1lLCBoZWxwIHRoZSBhdXRob3IgaW4gRDQxNDIxLjwvZGl2PjwvYmxvY2txdW90ZT48 ZGl2Pjxicj48L2Rpdj48ZGl2PkFoIHllcy4gSSd2ZSBiZWVuIGluIHRvdWNoIHdpdGggdGhlIGF1 dGhvciBmb3Igb3RoZXIgdGhpbmdzLCBhbmQgc29tZWhvdyB0aG91Z2h0IGl0IHdhcyB5b3UuLi4u ICBJJ2xsIHJlYWNoIG91dCB0byBoaW0gdmlhIG90aGVyIG1lYW5zLi4uPC9kaXY+PGRpdj48YnI+ PC9kaXY+PGRpdj5XYXJuZXI8L2Rpdj48L2Rpdj48L2Rpdj4NCg0KICAgICAgICA8L2Jsb2NrcXVv dGU+PGJyPg0KICAgIDwvZGl2Pg0KICAgICAgICA8L2Jsb2NrcXVvdGU+PGJyPg0KICAgIDwvZGl2 PjwvYmxvY2txdW90ZT48L2Rpdj4NCg0KICAgICAgICA8L2Jsb2NrcXVvdGU+PGJyPg0KICAgIDwv ZGl2Pg0KICAgICAgICA8L2Jsb2NrcXVvdGU+PGJyPg0KICAgIDwvZGl2PjwvYmxvY2txdW90ZT48 L2Rpdj4NCjwvYmxvY2txdW90ZT48L2Rpdj48L2Rpdj4NCg0KICAgICAgICA8L2Jsb2NrcXVvdGU+ PGJyPg0KICAgIDwvZGl2Pg== --b1=_LL69nXigG2smgSsejGdXqg0i84csaQ4J6T0lWPRUQ-- From nobody Sun Dec 8 17:41:05 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y5sjp17Z7z5gwXD for ; Sun, 08 Dec 2024 17:41:18 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y5sjn23Mlz40vh for ; Sun, 8 Dec 2024 17:41:17 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1035.google.com with SMTP id 98e67ed59e1d1-2eeb4d643a5so3289771a91.3 for ; Sun, 08 Dec 2024 09:41:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1733679676; x=1734284476; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=caM9jb9ruwkc2qYsw4n2bW0n+Hy4J9SbuOFk+U/0N6I=; b=wyu5+1snb9m+ZlyiOlEWEWz2rSnnymS0Y8Y4I85NYzmIE99JelBciHx2d74/KLet7H 8UjlhUA3auxK1hg8+nKQV4f9Xjm/3ahpvsxXxI2kdhkjP2lXZ83i+gqCKUBvdY8CMCKH K3qRn6I0vuVwiUtaQWWXZfzx1nbjGHLNI1Xgd9LT07JSpBorF8oDG2/+FAZmnCbPrmej j76eotWv2hXezz6rIbnUWvhQPwffFwiEVcSIwNwgkctJPHq/v8oy9rHE/5eRd/k5cK9+ zRJOZ7N0MXfK/g6h8OeuSBwHcwysyzLoV2P+heWxSWrlGCbW14cLSeXjj7rh1mlzB1kX huqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733679676; x=1734284476; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=caM9jb9ruwkc2qYsw4n2bW0n+Hy4J9SbuOFk+U/0N6I=; b=Et3KpODtvQY5hUTAHYkY/kffT5drZIq3j0cyDOa9XKjT00Mnboi2kz9ejYaI49JqHk EBdAJKPr8z+9V84/QARgXSDZfha8wTx/kzd66D60PmsIX+KmPD/Z/3cdq0Q/ygaYtNKb U62elB11gC049PLfkxReexpXNOAJ8XJo86veZXyaojd/yHyiqhdKFZW1uuhJOo2JH+Pd NsgedGR+Ihy2AcDw3WwTwJaWb7N5+0rygFZsWJc5ls3dWA8fT2N/0UwO4FtnQcflVx4V PiDYRkG8nyaSHPMWso5yyPSwu43ITZMQQSMbY+oR2q3C6l/oq5LCWVDgBjKcnjkzimVx 3XYQ== X-Forwarded-Encrypted: i=1; AJvYcCVnjixIE9bFzLDD/wLLn2s+pTAxdIH8o/F6f3H2uj0m1gtyqGjg6Poy2PN9zsWRb/h1+PIXgaDpTSHDpbu4xCk=@freebsd.org X-Gm-Message-State: AOJu0YxxPxEMzDlH034VLoQNedYO8Yqjnvv80Lxzwm3mnX7GUben9cr5 E10ET6j9FiIhCopLAc0vfrfQmWZtlbB8sOCtT75kOi6fyj3NAhsX9bUd3dtO85HwdTwwPYI80ef GaS6unJDI9AZR0r9LJdUAIXaomOKHOCIUIxWKOA== X-Gm-Gg: ASbGncv9gDzswBtyO30VxYUuU4r36OnC9ufvQQfVqmGY5GAdweI8q37EgDHFQBwzuOs JvFJUOmm+BsnDVxUqSghYY/SRXquyqVU= X-Google-Smtp-Source: AGHT+IFP7UJYkE65s1X7rNXdycBoIN5qXxFtjHlVDoI8JMhafe3CmvTlti39TUEr+zYKgEz1T7Zs4B8MH1+c6UXKDr4= X-Received: by 2002:a17:90b:4c04:b0:2ee:fdf3:390d with SMTP id 98e67ed59e1d1-2ef6ab06a5amr13293608a91.31.1733679676001; Sun, 08 Dec 2024 09:41:16 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <4L_wVuJx1yIMEv85fQKvrJp8QiaTK4Fe_TvymIq0vcdwdHqa06Ys4lqAM8aHb-kefxPiIZW7kxT8qI7hmv4bLngKUlzIWfBVzDcaz4VRIPY=@proton.me> In-Reply-To: <4L_wVuJx1yIMEv85fQKvrJp8QiaTK4Fe_TvymIq0vcdwdHqa06Ys4lqAM8aHb-kefxPiIZW7kxT8qI7hmv4bLngKUlzIWfBVzDcaz4VRIPY=@proton.me> From: Warner Losh Date: Sun, 8 Dec 2024 10:41:05 -0700 Message-ID: Subject: Re: Long time outdated jemalloc To: Minsoo Choo Cc: cglogic , FreeBSD CURRENT Content-Type: multipart/alternative; boundary="0000000000003e31c50628c5c16a" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Y5sjn23Mlz40vh X-Spamd-Bar: ---- --0000000000003e31c50628c5c16a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Great! I'll take a look at that, as well as do the merge the typical way (which only takes a few minutes now that I've bootstrapped things). I'll compare the two to see what diffs there might be (to act as a cross check for both methods). I'll then build a copy of the Netflix firmware with the change and put it on a couple machines and see if they can handle the load and if there are any performance regressions. I don't expect any, since malloc typically doesn't appear in the flame graphs as "visible", but you never know. So, once that's done, and I expect it to be done this week, I'll push it into main with both the proper vendor branch merge commits as well as an acknowledgement for this pull request and your work to move it forward (I'm just verifying the typical process will produce the same results and the typical process doesn't take a long time, etc). Warner On Sun, Dec 8, 2024 at 9:03=E2=80=AFAM Minsoo Choo wrote: > I resolved merge conflict by rebasing main. What's next? > > https://github.com/freebsd/freebsd-src/pull/1337 > On Sunday, December 1st, 2024 at 2:05 AM, Warner Losh > wrote: > > (sorry to follow up to my own email and topposting) > > I got the vendor branch bootstrapped: I created vendor/jemalloc and tagge= d > vendor/jemalloc/5.2.1, > and created a merge commit from that branch to main. I had to tweak the > FREEBSD-Xlist a little.I've not updated the other two FREEBSD-* files, bu= t > the steps were > documented in the commit messages (the vendor branch one is especially > long and > likely should migrate into the developer handbook). FREEBSD-update is > basically > a shell script to do the same thing that git subtree merge does, though > I'm sure some > tweaks could help the next time (if there is a next time, jemalloc > upstream seems to > have slowed way down of late). > > Next up,I'll create 5.3.0 import on the vendor branch, do the merge and > start testing (it will be > Minsoo's pull request, rebased, with any conflicts resolved and merge > commit recorded). > But that will have to be tomorrow or more likely during the work week. I'= m > too tired tonight > to get it right at the moment. > > And a special thanks to emaste for giving bz the right recipe for doing > the subtree merge > w/o git subtree for his work on the linux wifi drivers in the tree. > > Warner > > On Sat, Nov 30, 2024 at 9:38=E2=80=AFPM Warner Losh wrot= e: > >> Yea, I need to get a copy of jemalloc 5.3.0 and 5.2.1 to try to >> 'bootstrap' the vendor branch. >> Then I need to bootstrap it... >> >> I just did the same with edk2 (which had a vendor branch, but hadn't bee= n >> updated since svn times). >> However, jemalloc doesn't have a vendor branch yet, so I'll have to >> create that, but I'll start with the >> current version rather than doing full history... So I'll start there. >> I also just did awk and lua, so once I have things bootstrapped, I'll be >> able to add 5.3.0 and then layer Minsoo's on >> top of that and then start testing it somehow. >> >> Malloc makes me nervous to touch, honestly, but I'll give it a go and >> test boot on my system and >> maybe see if we can survive a workload at work w/o regressions... But I >> can't do a full test with lots >> of machines until after the first of the year (though I can do a couple >> for a few days before then). >> >> So my next step is to bootstrap the vendor branch... I'll give that a tr= y >> tonight. >> >> Warner >> >> On Sat, Nov 30, 2024 at 8:26=E2=80=AFPM Minsoo Choo >> wrote: >> >>> I have already submitted PR on github ( >>> https://github.com/freebsd/freebsd-src/pull/1337) and phabricator ( >>> https://reviews.freebsd.org/D41421). I don't have access (commit bit) >>> to freebsd git repo, so there is nothing I can do at this point since >>> vendor import and landing patches requires commit bit. >>> On Saturday, November 30th, 2024 at 1:42 PM, cglogic < >>> cglogic@protonmail.com> wrote: >>> >>> I see, it happens. >>> Maybe another committer will volunteer to do the update. >>> I hope it will make its way into 15.0 release. >>> >>> Thanks. >>> On Friday, November 29th, 2024 at 9:38 PM, Warner Losh >>> wrote: >>> >>> I've been swamped. we need to bootstrap the vendor branch, and the way >>> prior updates were done >>> isn't so great. >>> >>> Warner >>> >>> On Mon, Nov 25, 2024 at 2:21=E2=80=AFAM cglogic wrote: >>> >>>> Hello guys, >>>> >>>> How the update of jemalloc is going? It's November now. >>>> >>>> Thanks. >>>> On Monday, July 22nd, 2024 at 7:02 PM, Minsoo Choo < >>>> minsoochoo0122@proton.me> wrote: >>>> >>>> First, sorry for late response. >>>> >>>> cglogic, thank you for bringing up this issue again since I nearly >>>> forgot that this issue was still open. >>>> >>>> Warner, as I can't access to my FreeBSD instance until the end of >>>> August, but I can still edit and push the code through my Arm Mac. Thi= s >>>> means that I can't test the updated code on my machine, but I can join= the >>>> review process and listen to change proposals. >>>> >>>> I'll open a Github PR in a few hours. (The phabricator review will sta= y >>>> opened just in case) >>>> On Monday, July 22nd, 2024 at 5:08 AM, Warner Losh >>>> wrote: >>>> >>>> >>>> >>>> On Sun, Jul 21, 2024 at 2:03=E2=80=AFPM cglogic wrote: >>>> >>>>> >>>>> On Sunday, July 21st, 2024 at 6:54 AM, Warner Losh >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On Sat, Jul 20, 2024 at 1:59=E2=80=AFAM cglogic >>>>> wrote: >>>>> >>>>>> Hello FreeBSD community, >>>>>> >>>>>> After Jason Evans stepped aside from maintaining jemalloc in >>>>>> FreeBSD, it's not updating in time anymore. >>>>>> Version 5.3.0 was released May 6, 2022 and FreeBSD still not >>>>>> imported it into the tree. >>>>>> >>>>>> There is a pending review https://reviews.freebsd.org/D41421 from Au= g >>>>>> 11, 2023. >>>>>> I'm successfully running FreeBSD/amd64 system with D41421 applied fo= r >>>>>> 8 months, as well as many other people. >>>>>> >>>>>> Can it be reviewed and committed to CURRENT? >>>>>> Or, if there is no committers willing to do it, can commit bit be >>>>>> given to submitter or another person willing to do this? >>>>>> >>>>>> It's very disappointing when users spend their time to fill such gap= s >>>>>> and their efforts just ignored by the developers. >>>>>> Every year FreeBSD Community Survey asking about user experience in >>>>>> contributing to FreeBSD. >>>>>> Here you can see an example of such contributing. >>>>>> >>>>>> >>>>> First, thank you for being persistent and continuing to bring it up. >>>>> It's important to do that to make sure this (and your many other) >>>>> contribution doesn't fall on the floor. >>>>> >>>>> And to be fair, we're only 3 months since the last update. Still, >>>>> quite a bit longer than you should have to wait, but not nearly the y= ear >>>>> the original date suggests. >>>>> >>>>> And this is a perfect storm of "how the project is bad at accepting >>>>> contributions": >>>>> (1) The original submission was close to the 14 branch creation time. >>>>> This meant that we weren't well prepared to look at it since it is su= ch an >>>>> invasive change (at least on its surface). It also slowed the initial >>>>> response... >>>>> (2) There was a number of back and forth requests for changes, which >>>>> took time to sort out... >>>>> (3) The size of this is huge, well beyond the capacity of Phabricator >>>>> to review accurately... >>>>> (4) It's a vendor import. That means we can't just drop the >>>>> Phabricator review into the tree... >>>>> (5) It's phabricator: this is a great tool for developers, but we hav= e >>>>> a terrible track record of using it for intake from new contributors.= We >>>>> don't have any oversight at all over this tool, at there's at best te= pid >>>>> and luke warm attempts to look for drop balls. >>>>> >>>>> All of these things are a terrible experience. I can only apologize. >>>>> These days, we might steer this towards github, but the 'vendor impor= t' >>>>> means you really need someone on the inside, or you need to be on the >>>>> inside to make that work. >>>>> >>>>> So, how to move forward? Well, I'd like to propose the following: >>>>> (1) submit all the other Phabricator reviews you have open (they are >>>>> mostly good, or close to good) to github. Github is being actively ma= naged >>>>> and will make it faster to get things it. It's a much better tool for= new >>>>> contributors (and even frequent contributors of smallish things). >>>>> (2) I should do an vendor import of 5.3.0 from github, and do the >>>>> merge to a branch and push that to github. You can then layer on your >>>>> changes and those can be reviewed more closely as a pull request agai= nst >>>>> the branch I push. I suspect that most of the issues are sorted out a= lready >>>>> (3) I'll land it via that route... >>>>> >>>>> And, if the sum of the other pull requests and this are good (and I >>>>> suspect they will be), then we can talk about commit bits and such. >>>>> >>>>> It's experiences like this which is why I'm trying to stand up github >>>>> pull requests as a reliable way to get things and and the best place = to >>>>> send people... >>>>> >>>>> Thanks again for persisting, and also for expressing this criticism >>>>> that we (hopefully) can use to make it better. >>>>> >>>>> Warner >>>>> >>>>> >>>>> Hello. >>>>> >>>>> I'm not the author of D41421. Just applied the patch to test it 8 >>>>> months ago. And recently discovered that it's still not committed. >>>>> I can't copy your message to Phabricator because don't have an >>>>> account. Please, if you have time, help the author in D41421. >>>>> >>>> >>>> Ah yes. I've been in touch with the author for other things, and >>>> somehow thought it was you.... I'll reach out to him via other means..= . >>>> >>>> Warner >>>> >>>> >>>> >>>> >>> >>> > --0000000000003e31c50628c5c16a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Great! I'll take a look at that, as well as do the mer= ge the typical way (which only takes a few minutes now that I've bootst= rapped things). I'll compare the two to see what diffs there might be (= to act as a cross check for both methods). I'll then build a copy of th= e Netflix firmware with the change and put it on a couple machines and see = if they can handle the load and if there are any performance regressions. I= don't expect any, since malloc typically doesn't appear in the fla= me graphs as "visible", but you never know.

So= , once that's done, and I expect it to be done this week, I'll push= it into main with both the proper=C2=A0vendor branch merge commits as well= as an acknowledgement for this pull request and your work to move it forwa= rd (I'm just verifying the typical process will produce the same result= s and the typical process doesn't take a long time, etc).
Warner

On Sun, Dec 8, 2024 at 9:03= =E2=80=AFAM Minsoo Choo <min= soochoo0122@proton.me> wrote:
I resolved merge conflict by rebasing main. What&= #39;s next?

On Sunday, December 1st, 2024 at 2:05 AM, Warner Losh <imp@bsdimp.com> wrote:<= br>
(sorry to follow up to my own= email and topposting)

I got the vendo= r branch bootstrapped: I created vendor/jemalloc and tagged vendor/jemalloc= /5.2.1,
and created a merge commit from that branch to main. I ha= d to tweak the
FREEBSD-Xlist a little.I've not updated the ot= her two FREEBSD-* files, but the steps were
documented in the com= mit messages (the vendor branch one is especially long and
likely= should migrate into the developer handbook). FREEBSD-update is basically
a shell script to do the same thing that git subtree merge does, t= hough I'm sure some
tweaks could help the next time (if there= is a next time, jemalloc upstream seems to
have slowed way down = of late).

Next up,I'll create 5.3.0 import on = the vendor branch, do the merge and start testing (it will be
Min= soo's pull request, rebased, with any conflicts resolved and merge comm= it recorded).
But that will have to be tomorrow or more likely du= ring the work week. I'm too tired tonight
to get it right at = the moment.

And a special thanks to emaste for giv= ing bz the right recipe for doing the subtree merge
w/o git subtr= ee for his work on the linux wifi drivers in the tree.

=
Warner

On Sat, Nov 30, 2024 at 9:38=E2=80=AFPM Warner Losh <imp@bsdimp.com> wrote:
Yea, I need to get a copy of jemalloc= 5.3.0 and 5.2.1 to try to 'bootstrap' the vendor branch.
Then = I need to bootstrap it...

I just did the same with= edk2 (which had a vendor branch, but hadn't been updated since svn tim= es).
However, jemalloc doesn't have a vendor branch yet, so I= 'll have to create that, but I'll start with the
current = version rather than doing full history... So I'll start there.
I also just did awk and lua, so once I have things bootstrapped, I'l= l be able to add 5.3.0 and then layer Minsoo's on
top of tha= t and then start testing it somehow.

Malloc makes = me nervous to touch, honestly, but I'll give it a go and test boot on m= y system and
maybe see if we can survive a workload at work w/o r= egressions... But I can't do a full test with lots
of mac= hines until after the first of the year (though I can do a couple for a few= days before then).

So my next step is to bootstra= p the vendor branch... I'll give that a try tonight.

Warner

On Sat, Nov 30, 2024 at 8:26=E2=80=AFPM Minsoo Choo <= minsoochoo0122@proton.me> wrote:
I have already submitted PR on github (https://github.com/freebsd/freebsd-src/pull/1337) and phabricator (https://reviews.free= bsd.org/D41421). I don't have access (commit bit) to freebsd= git repo, so there is nothing I can do at this point since vendor import a= nd landing patches requires commit bit.
On Saturday, November 30th, 2024 at 1:42 PM, cglogic <cglogic@protonmail.com> wrote:
I se= e, it happens.
Maybe another committer will volunteer to do the update.I hope it will make its way into 15.0 release.

= Thanks.
On Friday, November 29th, 2024 at 9:38 PM, Warner Losh <imp@bsdimp.com> wrote:
I've been swamped. we need to bootstrap th= e vendor branch, and the way prior updates were done
isn't so great= .

Warner

On Mon, Nov 25, 2024 at 2:21=E2=80= =AFAM cglogic <cglogic@protonmail.com> wrot= e:
Hello guys,

How the update of jemalloc is go= ing? It's November now.

Thanks.
On Monday, July 22nd, 2024 at 7:02 PM, Minsoo Choo <minsoochoo0122@proton.me> wrote:
Firs= t, sorry for late response.

cglogic, thank you for bringing up this issue again since I near= ly forgot that this issue was still open.

Warner, as I can't access to my FreeBSD instan= ce until the end of August, but I can still edit and push the code through = my Arm Mac. This means that I can't test the updated code on my machine= , but I can join the review process and listen to change proposals.

I'll open a Github P= R in a few hours. (The phabricator review will stay opened just in case)
On Monday, July 22nd, 2024 at 5:08 AM, Warner Losh <imp@bsdimp.com> wrote:


On Sun, Jul 21, 2024 at 2= :03=E2=80=AFPM cglogic <cglogic@protonmail.com= > wrote:

On Sunday, July 21st, 2024 at 6:54 AM, Warner Losh <imp@bsdimp.com> wrote:


On Sat, Jul 20, 2024 at 1= :59=E2=80=AFAM cglogic <cglogic@protonmail.com= > wrote:
Hello FreeBSD community,

After Jason Evans stepped aside from = maintaining jemalloc in FreeBSD, it's not updating in time anymore.
Version 5.3.0 was released = May 6, 2022 and FreeBSD still not imported it into the tree.

There is a pending review https://reviews.freebsd.org/D41421 from Aug 11, 2023.
I'm succes= sfully running FreeBSD/amd64 system with D41421 applied for 8 months, as we= ll as many other people.

= Can it be reviewed and committed to CURRENT?
Or, if there is no committe= rs willing to do it, can commit bit be given to submitter or another person= willing to do this?
=
It's very disappointing when users spend their time to fi= ll such gaps and their efforts just ignored by the developers.
Every year FreeBSD Community Survey asking about user experience in contr= ibuting to FreeBSD.
Here you can see an example of su= ch contributing.


Firs= t, thank you for being persistent and continuing to bring it up. It's i= mportant to do that to make sure this (and your many other) contribution do= esn't fall on the floor.

And to be fair, w= e're only 3 months since the last update. Still, quite a bit longer tha= n you should have to wait, but not nearly the year the original date sugges= ts.

And this is a perfect storm of "how t= he project is bad at accepting contributions":
(1) The origi= nal submission was close to the 14 branch creation time. This meant that we= weren't well prepared to look at it since it is such an invasive chang= e (at least on its surface). It also slowed the initial response...
(2) There was a number of back and forth requests for changes, which= took time to sort out...
(3) The size of this is huge, well beyo= nd the capacity of Phabricator to review accurately...
(4) It'= ;s a vendor import. That means we can't just drop the Phabricator revie= w into the tree...
(5) It's phabricator: this is a great tool= for developers, but we have a terrible track record of using it for intake= from new contributors. We don't have any oversight at all over this to= ol, at there's at best tepid and luke warm attempts to look for drop ba= lls.

All of these things are a terrible experience= . I can only apologize. These days, we might steer this towards github, but= the 'vendor import' means you really need someone on the inside, o= r you need to be on the inside to make that work.

= So, how to move forward? Well, I'd like to propose the following:
=
(1) submit all the other Phabricator reviews you have open (they are m= ostly good, or close to good) to github. Github is being actively managed a= nd will make it faster to get things it. It's a much better tool for ne= w contributors (and even frequent contributors of smallish things).
(2) I should do an vendor import of 5.3.0 from github, and do the merge = to a branch and push that to github. You can then layer on your changes and= those can be reviewed more closely as a pull request against the branch I = push. I suspect that most of the issues are sorted out already
(3) I'll land it via that route...

And, if = the sum of the other pull requests and this are good (and I suspect they wi= ll be), then we can talk about commit bits and such.

It's experiences like this which is why I'm trying to stand up g= ithub pull requests as a reliable way to get things and and the best place = to send people...

Thanks again for persistin= g, and also for expressing this criticism that we (hopefully) can use to ma= ke it better.

Warner

Hello.

I'm not the author of = D41421. Just applied the patch to test it 8 months ago. And recently = discovered that it's still not committed.
I can't copy your message to Phabricator becaus= e don't have an account. Please, if you have time, help the auth= or in D41421.

Ah yes. I've been i= n touch with the author for other things, and somehow thought it was you...= . I'll reach out to him via other means...

Wa= rner





--0000000000003e31c50628c5c16a-- From nobody Sun Dec 8 19:30:36 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y5w806jNMz5f5B3 for ; Sun, 08 Dec 2024 19:30:40 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y5w806DB6z4Ljt; Sun, 8 Dec 2024 19:30:40 +0000 (UTC) (envelope-from ronald@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1733686240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SySV/E5AxfDCVTu/pKV9ex4imbByY32Wqj/mMFwj8bo=; b=fCQsmHLQnRP4qPFeHuuNPubpr2AqsvKHZHkqGjivWgznN0HW1g4d3F+aDiyGZ9oso7Ry0U DjBDByV+JCn+C4PZhA/4OUAl7D3Sm19TmCrw8LsdzeQCE1Um/WwJyFuMOHbtJaQM+XxFkQ 86U+FnLkE/TdQrsEU1k7X9TrjcQ1jraPUWJfMMygINERa+sqalmXNzr8IvOzHsKsvOTCzs h9wvs5qGQC5YclrU1rCLHBYV8WKIJHGDPYSNdM4VZJwk5Jj6rWFNdiyWp1PwiLmrKEGiFW QuUfNI//lioMruroV5H/NlkSg88QVQghR3RTvXRFIcwUuaUvMjUQKJAXMLF0CQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1733686240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SySV/E5AxfDCVTu/pKV9ex4imbByY32Wqj/mMFwj8bo=; b=xzpBHMHe0oepF/XSxa2OojwvZeAC18SZJ/xLlqzxK+LKrzg2Psmkov3BQaKkzxEobKiaTv SNaFyP4q1bE08DsSU0Nnu2va7b5nBcihUqTFJwc1Mi6hHU6S+GXtTZK4rp8PaOCDhG79B1 cVxq3q1W+JqcHGs1PmlyP/zsmUCgTrwq8u7MTdhsk1u8976AzaTcAb4Nuz48aizH+45BgD 6lI2Ab3lIIyxT8tlBxLBZCqKl7pnVtHEQNPCRRPhJ7Y1WAlXQJE7UaTrHFIhEN5UHPTu6R OmMjwnw9/oeMgk75+6fN59QttwdM/6ZC+wyb7hhvu4nhagsKJySrBqhQUKt0tQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1733686240; a=rsa-sha256; cv=none; b=nyaDBzlulA5g27gjxVnRa0Sy7+srQOXHRPU2Rvwx6A35Pp5qsz+bHo3Y8b9bg0onCbhEJk tFXpaxE6QPWZV0pCO4814+HLekCqqDmiYZkqaZh8dCG+/CQr1+lX6jMYj6yJn/pgGHZcKu KZU5gNhTt25Cx1gVrtJYadwr1n1v625KgvPO73m8HG1wqi+svSasnngOZKbOvZdE3RA2U9 7b2HezkJpvT3DnzNtcKw/F7fzuFoTRSgBBAsWqJxhSgnewvCKth7zsYboPz7AVI4oYuzBw kYwerQFt859D9aob67tyAtzMmJDkOsoEyafQDr8qL1ERRBeFw1Vfyfb2mlc7kA== Received: from [IPV6:2001:1c00:2709:2010:c129:38:42e1:4c97] (2001-1c00-2709-2010-c129-0038-42e1-4c97.cable.dynamic.v6.ziggo.nl [IPv6:2001:1c00:2709:2010:c129:38:42e1:4c97]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: ronald/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Y5w803Zmwz1Ldx; Sun, 8 Dec 2024 19:30:40 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Message-ID: Date: Sun, 8 Dec 2024 20:30:36 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: (ipfw) Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out To: FreeBSD User Cc: freebsd-current@freebsd.org References: <20241206034709.4dd32cc5@thor.intern.walstatt.dynvpn.de> <279848701.11738.1733510402875@localhost> <20241206210947.3ae835e4@thor.intern.walstatt.dynvpn.de> Content-Language: en-US From: Ronald Klop In-Reply-To: <20241206210947.3ae835e4@thor.intern.walstatt.dynvpn.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi, I can reproduce your error. Today I updated my RPI4 from a build of Oct 23 to Dec 6. And I can reproduce the problem. After about 2 hours scp exits with: client_loop: send disconnect: Broken pipe scp: Connection closed Working: FreeBSD rpi4 15.0-CURRENT FreeBSD 15.0-CURRENT #4 main-d2e7bb630b8-dirty: Wed Oct 23 00:55:12 CEST 2024 ronald@rpi4:/data/ronald/freebsd/obj/data/ronald/freebsd/src/main/arm64.aarch64/sys/GENERIC-NODEBUG arm64 Broken: FreeBSD rpi4 15.0-CURRENT FreeBSD 15.0-CURRENT #5 main-839fb85336a-dirty: Sat Dec 7 22:33:27 CET 2024 ronald@rpi4:/data/ronald/freebsd/obj/data/ronald/freebsd/src/main/arm64.aarch64/sys/GENERIC-NODEBUG arm64 A cronjob which does a scp to another server didn't work anymore. When I go back to the previous BE it works fine again. Ipfw disable firewall also makes the scp work. Scp also seems to work fine if I replace the statefull firewall rules with stateless "pass all from any to any". Regards, Ronald. Op 06-12-2024 om 21:09 schreef FreeBSD User: > Am Fri, 6 Dec 2024 19:40:02 +0100 (CET) > Ronald Klop schrieb: > >> Might be useful to share your ipfw config. > > Sorry, my posting must have been disturbing (having in mind a "deny any rule and then > disabling the FW ...). > > Well, the IPFW setup itself is explained quickly - I use almost the vanilla rc.conf-issued > IPFW (settings: firewall_type="workstation", firewall_logif="YES", > firewall_myservices="22/tcp", firewall_allowservices="any"). The hosts in question have the > following kernel configuration, I provide the option tags that might be of interest or, if > not, just for the record, as they are not part of GENERIC, see below. > > Also, I'll provide some sysctl setting performed via /etc/sysctl.conf.local, see below. > > The configuration and settings have been mostly unchanged over a couple of months for now and > did not induce trouble so far. > > As it deemed fit regarding time and my limited skills, I disabled and enabled piece by piece > of the MAC_ and NETGRAPH_ options - without any success so far - my "measurement" is fetching > emails via claws-mail (all TLS). claws-mail reports "corrupted/broken stream", does have > authetication issues and is de facto unusable - it doesn't refresh IMAP based email fetches > and doesn't even quit without a hard kill. > Another "indicator" is the time taken to "git pull" of ZFS filesystems: cloning and pulling > takes unusual long (/usr/src is UFS/FFS, /usr/ports on a ZFS pool and since the problem > occured, it makes a mutual difference). > > While git pull or clone mutually stuck and claws-mail is endlessly fetching/authenticating > emails and never responding back in a usable manner, performing > > "ipfw disable firewall" > > makes all of a sudden the system work again as usual and expected. > > As reported - the problem spreads across all of my CURRENT hosts as I'm going to update them > towards a recent CURRENT (they all share similar static kernel configs as described here). Most > of the boxes do not show the weird reluctant behaviour when pulling via git, but weren't > capable of cloning, bailing out with the timeout reported earlier. > > I use one CURRENT box as my personal desktop, so no other (server) CURRENT show the Email > problem in detail as described. > > And, for the record: I haven't commented out the "options IPFIREWALL" yet in the kernel > config ... > > > Kind regards > > oh > > [ KERNEL config different from vanilla GENERIC ] > > options RATELIMIT > options ZFS > options TCPHPTS > options MROUTING > options IPSEC > options SCTP > > options MAC_BSDEXTENDED > options MAC_PORTACL > options MAC_IPACL > options MAC_NTPD > #options MAC_DO > > options NETGRAPH > options NETGRAPH_IPFW > options NETGRAPH_ETHER > options NETGRAPH_EIFACE > options NETGRAPH_VLAN > #options NETGRAPH_NAT > options NETGRAPH_DEVICE > #options NETGRAPH_PPPOE > options NETGRAPH_SOCKET > options NETGRAPH_KSOCKET > options NETGRAPH_NETFLOW > #options NETGRAPH_CAR > > # IPFW firewall > options IPFIREWALL > options IPFIREWALL_VERBOSE > options DUMMYNET # traffic shaper > > options BPF_JITTER # adds support for BPF just-in-time compiler. > > # Pseudo devices not in GENERIC. > device enc # IPsec device > device stf # 6to4 IPv6 over IPv4 encapsulation > device carp # Common address redundancy protocol > device lagg # Link aggregation > device gre # GRE Tunnel > device epair # A pair of virtual back-to-back connected Ethernet interfaces > device if_bridge # bridge device > device vxlan # Virtual eXtensible LAN interface > > > For the MAC_ Modules: the appropriate OIDs (sysctl) are disabled as far as the MAC module > influence the initial behaviour if unconfigured, for instance > (/etc/sysctl.conf.local) > > [ /etc/sysctl.conf.local ] > security.mac.bsdextended.enabled=0 > security.mac.mls.enabled=0 > security.mac.portacl.enabled=0 > security.mac.do.enabled=0 > security.mac.ipacl.ipv6=0 > security.mac.ipacl.ipv4=0 > # > net.bpf.optimize_writers=1 > # > net.inet.ip.fw.verbose=1 > #net.inet.ip.fw.verbose_limit=10 > net.inet.ip.fw.dyn_keep_states=1 > > > > > >> >> Van: FreeBSD User >> Datum: 6 december 2024 03:47 >> Aan: freebsd-current@freebsd.org, freebsd-ipfw@freebsd.org >> Onderwerp: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: >> Operation timed out >> >>> >>> >>> Am Thu, 5 Dec 2024 17:33:54 +0100 >>> FreeBSD User schrieb: >>> >>> I found the culprit! >>> >>> Disabling IPFW ("ipfw disable firewall") turns system back to normal! >>> >>> For the record: on recent CURRENT, since approx. Nov. 30 and/or December 1st CURRENT seems >>> to corrupt network connections. >>> >>> IPFW is compiled statically into the kernel. >>> >>> The problem sketched below can be reproduced in a more or less obvious manner on recent >>> CURRENT: git pull/git clone of a regular FreeBSD source repo or ports via git+https takes >>> either a couple of time (up to several mintes to initiate the pull) - or, in some worse >>> cases here, the box runs into >>> error: RPC failed; curl 56 Recv failure: Operation timed out >>> >>> claws-mail complains about "corrupted/broken stream", fetching emails takes Aeons - >>> forever, the client does not come back even after several hours. >>> >>>> On Thu, 5 Dec 2024 16:55:00 +0100 >>>> Daniel Tameling wrote: >>>> >>>>> On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote: >>>>>> On Wed, 04 Dec 2024 17:20:39 +0000 >>>>>> "Dave Cottlehuber" wrote: >>>>>> >>>>>> Thank you very much for responding! >>>>>> >>>>>>> On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: >>>>>>>> On most recent CURRENT (on some boxes of ours, not all) fetch/git seem >>>>>>>> to be stuck >>>>>>>> forever fetching tarballs from ports, fetching Emails via claws-mail >>>>>>>> (TLS), opening >>>>>>>> websites via librewolf and firefox or pulling repositories via git. >>>>>>>> >>>>>>>> CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 >>>>>>>> 23:11:07 CET 2024 >>>>>>>> amd64 >>>>>>>> >>>>>>>> When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: >>>>>>>> >>>>>>>> error: RPC failed: curl 56 recv failure: Operation timed out >>>>>>> >>>>>>> Generally it would be worth seeing if the HTTP(S) layers are doing the right thing >>>>>>> or not, and then working down from there, to tcpdump / wireshark and then if >>>>>>> necessary into kernel itself. >>>>>> >>>>>> My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and >>>>>> theory,of course). I tried due to "a feeling" my used older Intel based NIC could >>>>>> have some checksum issues like in the past (I saw e1000 driver updates recently >>>>>> flowing into FreeBSD CURRENT). >>>>>>> >>>>>>> If fetch fails reliably in ports distfile fetching, then isolate a suitable >>>>>>> tarball, and try it again in curl, with tcpdump already prepared to capture >>>>>>> traffic to the remote host. >>>>>>> >>>>>>> tcpdump -w /tmp/curl.pcap -i ... host ... >>>>>>> >>>>>>> env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace >>>>>>> /tmp/curl.log https://what.ev/er >>>>>>> >>>>>>> I would guess that between the two something useful should pop up. >>>>>>> >>>>>>> I like opening the pcap in wireshark, it often has angry red and black highlighted >>>>>>> lines already giving me a hint. >>>>>>> >>>>>>> The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS >>>>>>> traffic as well in case there are issues further in. Very handy, >>>>>>> see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. >>>>>>> >>>>>>> If your issues only occur with git pull, its also curl inside and supports similar >>>>>>> debugging. Ferreting >>>>>>> through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. >>>>>>> >>>>>>> A+ >>>>>>> Dave >>>>>>> >>>>>> >>>>>> Thanks for the hints and precious tips! I'll digg deeper into the matter. >>>>>> >>>>>> In the meanwhile, I updated some other machines running CURRENT since approx. two >>>>>> weeks with an older CURRENT to the most recent one - and face similar but not >>>>>> identical problems! >>>>>> Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems >>>>>> except they take longer to accomplish than expected. >>>>>> Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout. >>>>>> >>>>>> On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD >>>>>> 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), >>>>>> cloning attempts for 14.2-RELENG ends up in this mess: >>>>>> >>>>>> # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/ >>>>>> Cloning into '14.2-RELENG/src'... >>>>>> error: RPC failed; curl 56 Recv failure: Operation timed out >>>>>> fatal: expected 'packfile' >>>>>> >>>>>> This is nasty. The host now in question has an i350 based dual-port NIC - the host's >>>>>> kernel is very similar to the box I reported the issue first time, both do have >>>>>> customized kernels (in most cases, I compile several modules like ZFS and >>>>>> several NETGRAPH modules statically into the kernel - a habit inherited from a small >>>>>> FBSD project I configured (I wouldn't say developed) which does not allow loadable >>>>>> kernel modules due to regulations. >>>>>> >>>>>> I hoped others would stumble over this tripwire in recent CURRENT sources, since the >>>>>> phenomena and its distribution over a bunch of CURRENT boxes with different OS states >>>>>> seemingly show different behviour. >>>>>> >>>>>> And for the record: I also build my ports via poudriere and mostly via make. I also >>>>>> rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and >>>>>> so on to ensure having latest sources/packages. >>>>>> >>>>>> (I repeat myself here again, sorry, its for the record). >>>>>> >>>>>> Will report in on further development and "investigations" >>>>>> >>>>>> Kind regards and thanks, >>>>>> >>>>>> oh >>>>>> >>>>>> >>>>> >>>>> This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some >>>>> networking issues that got fixed later. >>>> >>>> No, pure Hardware and FreeBSD ... >>>> >>>>> >>>>> Otherwise I would start with ping and traceroute to figure out if they show this issue >>>>> and where it occurs. >>>>> >>>> >>>> >>> >>> >>> >>> -- >>> O. Hartmann >>> >>> >>> >>> >>> > > >