From nobody Fri Dec 6 02:46:42 2024 X-Original-To: freebsd-ipfw@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y4FzB2zqvz5g7jl; Fri, 06 Dec 2024 02:47:18 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp052.goneo.de (smtp5.goneo.de [IPv6:2001:1640:5::8:30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y4Fz82G6bz46Wx; Fri, 6 Dec 2024 02:47:16 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=mgJS8hCF; spf=pass (mx1.freebsd.org: domain of freebsd@walstatt-de.de designates 2001:1640:5::8:30 as permitted sender) smtp.mailfrom=freebsd@walstatt-de.de; dmarc=none Received: from hub1.goneo.de (hub1.goneo.de [IPv6:2001:1640:5::8:52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id 7C7C7240E6F; Fri, 6 Dec 2024 03:47:12 +0100 (CET) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id 9F0822402D6; Fri, 6 Dec 2024 03:47:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1733453230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XdFGH3CMbtptjDEBjebyi+V9iZ7X3JTk/OqwmFdpgmU=; b=mgJS8hCFJs1ZRlboyR+Q++xSfQLhEpvyRnRoSANQv5OWwz/TvNkOO054tFMj8t9BevhgIa WeIN37r3sw/yFZ7TYvMwie+0NGrXVViMNVFSaty/5M230CjUREQg3uUHAecOqs/3NnxlEW exa5ZVkazLJShybzuTnxiYhgE/QbOctY1US+ZvS7j1/ISKFQ6YH73XQwiq3gBW4vQZlfoe JcJxx+nVT5c9jZZziwMxb4wrrNRqn0I/4+7qDSwRayAcqIdsSwKuGQ8EnlMDLXTfW77Srh PcHFPr9KTiJTlu7Ne39chbzjmgG99KyuH5bFRVBAPyR8G59e14VKqkt6HqM6zw== Received: from thor.intern.walstatt.dynvpn.de (dynamic-078-054-013-094.78.54.pool.telefonica.de [78.54.13.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id 56A6024029B; Fri, 6 Dec 2024 03:47:10 +0100 (CET) Date: Fri, 6 Dec 2024 03:46:42 +0100 From: FreeBSD User To: freebsd-current@freebsd.org, freebsd-ipfw@freebsd.org Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: <20241206034709.4dd32cc5@thor.intern.walstatt.dynvpn.de> In-Reply-To: <20241205173354.23c4e592@hermann.intern.walstatt.dynvpn.de> References: <20241203204609.68e04364@hermann.dmz.walstatt.dynvpn.de> <6626e5c0-ba01-4966-a28c-82a25251ca3f@app.fastmail.com> <20241205115103.1ed6d7f6@hermann.intern.walstatt.dynvpn.de> <20241205173354.23c4e592@hermann.intern.walstatt.dynvpn.de> Organization: walstatt-de.de List-Id: IPFW Technical Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-ipfw List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ipfw@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: a8d101 X-Rspamd-UID: 61e20b X-Spamd-Result: default: False [-3.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; R_SPF_ALLOW(-0.20)[+ip6:2001:1640:5::8:0/112]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[2001:1640:5::8:30:from]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ORG_HEADER(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:25394, ipnet:2001:1640::/32, country:DE]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org,freebsd-ipfw@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[walstatt-de.de]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; DKIM_TRACE(0.00)[walstatt-de.de:+] X-Rspamd-Queue-Id: 4Y4Fz82G6bz46Wx X-Spamd-Bar: --- Am Thu, 5 Dec 2024 17:33:54 +0100 FreeBSD User schrieb: I found the culprit! Disabling IPFW ("ipfw disable firewall") turns system back to normal! For the record: on recent CURRENT, since approx. Nov. 30 and/or December 1st CURRENT seems to corrupt network connections. IPFW is compiled statically into the kernel. The problem sketched below can be reproduced in a more or less obvious manner on recent CURRENT: git pull/git clone of a regular FreeBSD source repo or ports via git+https takes either a couple of time (up to several mintes to initiate the pull) - or, in some worse cases here, the box runs into error: RPC failed; curl 56 Recv failure: Operation timed out claws-mail complains about "corrupted/broken stream", fetching emails takes Aeons - forever, the client does not come back even after several hours. > On Thu, 5 Dec 2024 16:55:00 +0100 > Daniel Tameling wrote: > > > On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote: > > > On Wed, 04 Dec 2024 17:20:39 +0000 > > > "Dave Cottlehuber" wrote: > > > > > > Thank you very much for responding! > > > > > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: > > > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem > > > > > to be stuck > > > > > forever fetching tarballs from ports, fetching Emails via claws-mail > > > > > (TLS), opening > > > > > websites via librewolf and firefox or pulling repositories via git. > > > > > > > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > > > > > 23:11:07 CET 2024 > > > > > amd64 > > > > > > > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: > > > > > > > > > > error: RPC failed: curl 56 recv failure: Operation timed out > > > > > > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing > > > > or not, and then working down from there, to tcpdump / wireshark and then if > > > > necessary into kernel itself. > > > > > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and > > > theory,of course). I tried due to "a feeling" my used older Intel based NIC could have > > > some checksum issues like in the past (I saw e1000 driver updates recently flowing > > > into FreeBSD CURRENT). > > > > > > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable tarball, > > > > and try it again in curl, with tcpdump already prepared to capture traffic to the > > > > remote host. > > > > > > > > tcpdump -w /tmp/curl.pcap -i ... host ... > > > > > > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace > > > > /tmp/curl.log https://what.ev/er > > > > > > > > I would guess that between the two something useful should pop up. > > > > > > > > I like opening the pcap in wireshark, it often has angry red and black highlighted > > > > lines already giving me a hint. > > > > > > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS > > > > traffic as well in case there are issues further in. Very handy, > > > > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. > > > > > > > > If your issues only occur with git pull, its also curl inside and supports similar > > > > debugging. Ferreting > > > > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. > > > > > > > > A+ > > > > Dave > > > > > > > > > > Thanks for the hints and precious tips! I'll digg deeper into the matter. > > > > > > In the meanwhile, I updated some other machines running CURRENT since approx. two > > > weeks with an older CURRENT to the most recent one - and face similar but not > > > identical problems! > > > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems > > > except they take longer to accomplish than expected. > > > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout. > > > > > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD > > > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), cloning > > > attempts for 14.2-RELENG ends up in this mess: > > > > > > # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/ > > > Cloning into '14.2-RELENG/src'... > > > error: RPC failed; curl 56 Recv failure: Operation timed out > > > fatal: expected 'packfile' > > > > > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's > > > kernel is very similar to the box I reported the issue first time, both do have > > > customized kernels (in most cases, I compile several modules like ZFS and > > > several NETGRAPH modules statically into the kernel - a habit inherited from a small > > > FBSD project I configured (I wouldn't say developed) which does not allow loadable > > > kernel modules due to regulations. > > > > > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the > > > phenomena and its distribution over a bunch of CURRENT boxes with different OS states > > > seemingly show different behviour. > > > > > > And for the record: I also build my ports via poudriere and mostly via make. I also > > > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and > > > so on to ensure having latest sources/packages. > > > > > > (I repeat myself here again, sorry, its for the record). > > > > > > Will report in on further development and "investigations" > > > > > > Kind regards and thanks, > > > > > > oh > > > > > > > > > > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some > > networking issues that got fixed later. > > No, pure Hardware and FreeBSD ... > > > > > Otherwise I would start with ping and traceroute to figure out if they show this issue > > and where it occurs. > > > > -- O. Hartmann