Date: Fri, 6 Dec 2024 19:40:02 +0100 (CET) From: Ronald Klop <ronald-lists@klop.ws> To: FreeBSD User <freebsd@walstatt-de.de> Cc: freebsd-ipfw@freebsd.org, freebsd-current@freebsd.org Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: <279848701.11738.1733510402875@localhost> In-Reply-To: <20241206034709.4dd32cc5@thor.intern.walstatt.dynvpn.de>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_11737_1079703555.1733510402870
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Might be useful to share your ipfw config.
Van: FreeBSD User <freebsd@walstatt-de.de>
Datum: 6 december 2024 03:47
Aan: freebsd-current@freebsd.org, freebsd-ipfw@freebsd.org
Onderwerp: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out
>
>
> Am Thu, 5 Dec 2024 17:33:54 +0100
> FreeBSD User schrieb:
>
> I found the culprit!
>
> Disabling IPFW ("ipfw disable firewall") turns system back to normal!
>
> For the record: on recent CURRENT, since approx. Nov. 30 and/or December 1st CURRENT seems to
> corrupt network connections.
>
> IPFW is compiled statically into the kernel.
>
> The problem sketched below can be reproduced in a more or less obvious manner on recent
> CURRENT: git pull/git clone of a regular FreeBSD source repo or ports via git+https takes
> either a couple of time (up to several mintes to initiate the pull) - or, in some worse cases
> here, the box runs into
> error: RPC failed; curl 56 Recv failure: Operation timed out
>
> claws-mail complains about "corrupted/broken stream", fetching emails takes Aeons - forever,
> the client does not come back even after several hours.
>
> > On Thu, 5 Dec 2024 16:55:00 +0100
> > Daniel Tameling wrote:
> >
> > > On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote:
> > > > On Wed, 04 Dec 2024 17:20:39 +0000
> > > > "Dave Cottlehuber" wrote:
> > > >
> > > > Thank you very much for responding!
> > > >
> > > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote:
> > > > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem
> > > > > > to be stuck
> > > > > > forever fetching tarballs from ports, fetching Emails via claws-mail
> > > > > > (TLS), opening
> > > > > > websites via librewolf and firefox or pulling repositories via git.
> > > > > >
> > > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2
> > > > > > 23:11:07 CET 2024
> > > > > > amd64
> > > > > >
> > > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes:
> > > > > >
> > > > > > error: RPC failed: curl 56 recv failure: Operation timed out
> > > > >
> > > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing
> > > > > or not, and then working down from there, to tcpdump / wireshark and then if
> > > > > necessary into kernel itself.
> > > >
> > > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and
> > > > theory,of course). I tried due to "a feeling" my used older Intel based NIC could have
> > > > some checksum issues like in the past (I saw e1000 driver updates recently flowing
> > > > into FreeBSD CURRENT).
> > > > >
> > > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball,
> > > > > and try it again in curl, with tcpdump already prepared to capture traffic to the
> > > > > remote host.
> > > > >
> > > > > tcpdump -w /tmp/curl.pcap -i ... host ...
> > > > >
> > > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace
> > > > > /tmp/curl.log https://what.ev/er
> > > > >
> > > > > I would guess that between the two something useful should pop up.
> > > > >
> > > > > I like opening the pcap in wireshark, it often has angry red and black highlighted
> > > > > lines already giving me a hint.
> > > > >
> > > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS
> > > > > traffic as well in case there are issues further in. Very handy,
> > > > > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that.
> > > > >
> > > > > If your issues only occur with git pull, its also curl inside and supports similar
> > > > > debugging. Ferreting
> > > > > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info.
> > > > >
> > > > > A+
> > > > > Dave
> > > > >
> > > >
> > > > Thanks for the hints and precious tips! I'll digg deeper into the matter.
> > > >
> > > > In the meanwhile, I updated some other machines running CURRENT since approx. two
> > > > weeks with an older CURRENT to the most recent one - and face similar but not
> > > > identical problems!
> > > > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems
> > > > except they take longer to accomplish than expected.
> > > > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout.
> > > >
> > > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD
> > > > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), cloning
> > > > attempts for 14.2-RELENG ends up in this mess:
> > > >
> > > > # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/
> > > > Cloning into '14.2-RELENG/src'...
> > > > error: RPC failed; curl 56 Recv failure: Operation timed out
> > > > fatal: expected 'packfile'
> > > >
> > > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's
> > > > kernel is very similar to the box I reported the issue first time, both do have
> > > > customized kernels (in most cases, I compile several modules like ZFS and
> > > > several NETGRAPH modules statically into the kernel - a habit inherited from a small
> > > > FBSD project I configured (I wouldn't say developed) which does not allow loadable
> > > > kernel modules due to regulations.
> > > >
> > > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the
> > > > phenomena and its distribution over a bunch of CURRENT boxes with different OS states
> > > > seemingly show different behviour.
> > > >
> > > > And for the record: I also build my ports via poudriere and mostly via make. I also
> > > > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and
> > > > so on to ensure having latest sources/packages.
> > > >
> > > > (I repeat myself here again, sorry, its for the record).
> > > >
> > > > Will report in on further development and "investigations"
> > > >
> > > > Kind regards and thanks,
> > > >
> > > > oh
> > > >
> > > >
> > >
> > > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some
> > > networking issues that got fixed later.
> >
> > No, pure Hardware and FreeBSD ...
> >
> > >
> > > Otherwise I would start with ping and traceroute to figure out if they show this issue
> > > and where it occurs.
> > >
> >
> >
>
>
>
> --
> O. Hartmann
>
>
>
>
>
------=_Part_11737_1079703555.1733510402870
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit
<html><head></head><body>Might be useful to share your ipfw config. <div><br><p><small><strong>Van:</strong> FreeBSD User <freebsd@walstatt-de.de><br><strong>Datum:</strong> 6 december 2024 03:47<br><strong>Aan:</strong> freebsd-current@freebsd.org, freebsd-ipfw@freebsd.org<br><strong>Onderwerp:</strong> Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out<br></small></p><blockquote style="margin-left: 5px; border-left: 3px solid #ccc; margin-right: 0px; padding-left: 5px;"><div class="MessageRFC822Viewer do_not_remove" id="P"><!-- P -->
<!-- processMimeMessage --><div class="TextPlainViewer do_not_remove" id="P.P"><!-- P.P -->Am Thu, 5 Dec 2024 17:33:54 +0100<br>
FreeBSD User <freebsd@walstatt-de.de> schrieb:<br>
<br>
I found the culprit!<br>
<br>
Disabling IPFW ("ipfw disable firewall") turns system back to normal!<br>
<br>
For the record: on recent CURRENT, since approx. Nov. 30 and/or December 1st CURRENT seems to<br>
corrupt network connections.<br>
<br>
IPFW is compiled statically into the kernel.<br>
<br>
The problem sketched below can be reproduced in a more or less obvious manner on recent<br>
CURRENT: git pull/git clone of a regular FreeBSD source repo or ports via git+https takes<br>
either a couple of time (up to several mintes to initiate the pull) - or, in some worse cases<br>
here, the box runs into <br>
error: RPC failed; curl 56 Recv failure: Operation timed out<br>
<br>
claws-mail complains about "corrupted/broken stream", fetching emails takes Aeons - forever,<br>
the client does not come back even after several hours.<br>
<br>
> On Thu, 5 Dec 2024 16:55:00 +0100<br>
> Daniel Tameling <tamelingdaniel@gmail.com> wrote:<br>
> <br>
> > On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote: <br>
> > > On Wed, 04 Dec 2024 17:20:39 +0000<br>
> > > "Dave Cottlehuber" <dch@skunkwerks.at> wrote:<br>
> > > <br>
> > > Thank you very much for responding!<br>
> > > <br>
> > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: <br>
> > > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem <br>
> > > > > to be stuck<br>
> > > > > forever fetching tarballs from ports, fetching Emails via claws-mail <br>
> > > > > (TLS), opening<br>
> > > > > websites via librewolf and firefox or pulling repositories via git.<br>
> > > > ><br>
> > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 <br>
> > > > > 23:11:07 CET 2024<br>
> > > > > amd64<br>
> > > > ><br>
> > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes:<br>
> > > > ><br>
> > > > > error: RPC failed: curl 56 recv failure: Operation timed out <br>
> > > > <br>
> > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing<br>
> > > > or not, and then working down from there, to tcpdump / wireshark and then if<br>
> > > > necessary into kernel itself. <br>
> > > <br>
> > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and<br>
> > > theory,of course). I tried due to "a feeling" my used older Intel based NIC could have<br>
> > > some checksum issues like in the past (I saw e1000 driver updates recently flowing<br>
> > > into FreeBSD CURRENT). <br>
> > > > <br>
> > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball,<br>
> > > > and try it again in curl, with tcpdump already prepared to capture traffic to the<br>
> > > > remote host.<br>
> > > > <br>
> > > > tcpdump -w /tmp/curl.pcap -i ... host ...<br>
> > > > <br>
> > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace<br>
> > > > /tmp/curl.log <a href="https://what.ev/er">https://what.ev/er</a><br>;
> > > > <br>
> > > > I would guess that between the two something useful should pop up.<br>
> > > > <br>
> > > > I like opening the pcap in wireshark, it often has angry red and black highlighted<br>
> > > > lines already giving me a hint.<br>
> > > > <br>
> > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS<br>
> > > > traffic as well in case there are issues further in. Very handy,<br>
> > > > see <a href="https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html">https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html</a>; for how to do that.<br>
> > > > <br>
> > > > If your issues only occur with git pull, its also curl inside and supports similar<br>
> > > > debugging. Ferreting<br>
> > > > through <a href="https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711">https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711</a>; should get you similar info.<br>
> > > > <br>
> > > > A+<br>
> > > > Dave<br>
> > > > <br>
> > > <br>
> > > Thanks for the hints and precious tips! I'll digg deeper into the matter.<br>
> > > <br>
> > > In the meanwhile, I updated some other machines running CURRENT since approx. two<br>
> > > weeks with an older CURRENT to the most recent one - and face similar but not<br>
> > > identical problems!<br>
> > > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems<br>
> > > except they take longer to accomplish than expected.<br>
> > > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout.<br>
> > > <br>
> > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD<br>
> > > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), cloning<br>
> > > attempts for 14.2-RELENG ends up in this mess:<br>
> > > <br>
> > > # git clone --branch releng/14.2 <a href="https://git.freebsd.org/src.git">https://git.freebsd.org/src.git</a>; 14.2-RELENG/src/<br>
> > > Cloning into '14.2-RELENG/src'...<br>
> > > error: RPC failed; curl 56 Recv failure: Operation timed out<br>
> > > fatal: expected 'packfile'<br>
> > > <br>
> > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's<br>
> > > kernel is very similar to the box I reported the issue first time, both do have<br>
> > > customized kernels (in most cases, I compile several modules like ZFS and<br>
> > > several NETGRAPH modules statically into the kernel - a habit inherited from a small<br>
> > > FBSD project I configured (I wouldn't say developed) which does not allow loadable<br>
> > > kernel modules due to regulations.<br>
> > > <br>
> > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the<br>
> > > phenomena and its distribution over a bunch of CURRENT boxes with different OS states<br>
> > > seemingly show different behviour.<br>
> > > <br>
> > > And for the record: I also build my ports via poudriere and mostly via make. I also<br>
> > > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and<br>
> > > so on to ensure having latest sources/packages.<br>
> > > <br>
> > > (I repeat myself here again, sorry, its for the record).<br>
> > > <br>
> > > Will report in on further development and "investigations" <br>
> > > <br>
> > > Kind regards and thanks,<br>
> > > <br>
> > > oh<br>
> > > <br>
> > > <br>
> > <br>
> > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some<br>
> > networking issues that got fixed later. <br>
> <br>
> No, pure Hardware and FreeBSD ...<br>
> <br>
> > <br>
> > Otherwise I would start with ping and traceroute to figure out if they show this issue<br>
> > and where it occurs.<br>
> > <br>
> <br>
> <br>
<br>
<br>
<br>
-- <br>
O. Hartmann<br>
<br>
</dch@skunkwerks.at></tamelingdaniel@gmail.com></freebsd@walstatt-de.de></div><!-- TextPlainViewer -->
<hr>
</div><!-- MessageRFC822Viewer -->
</blockquote><br><br><br></div></body></html>
------=_Part_11737_1079703555.1733510402870--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?279848701.11738.1733510402875>