From nobody Fri Dec 6 20:09:20 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y4j691dMQz5fp8J; Fri, 06 Dec 2024 20:09:53 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp052.goneo.de (smtp052.goneo.de [85.220.129.60]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y4j685hScz564g; Fri, 6 Dec 2024 20:09:52 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Authentication-Results: mx1.freebsd.org; none Received: from hub1.goneo.de (hub1.goneo.de [85.220.129.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id 5D544240D18; Fri, 6 Dec 2024 21:09:50 +0100 (CET) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id 74DE2240562; Fri, 6 Dec 2024 21:09:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1733515788; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9tlVOqP5LtBTwqnsT9G2I7CBMXXcRs9yIPWa6+WZImA=; b=lczPX0ODEy7JGetelVShwKORM1oNWwtcFbgfBKY/66Jf5pwnIzRP//E8EJSF43nfyXomtm hdE7DFTVFf6T2GtwJeLgiEZx3DEVooNOGpbcBdItGMgJBgjQZNL0izTtT/gOTVlTPDQcIO R7juuYScoOkIwleOLHS45QGLFHAanTuIHhmszh4hbsl/lFWsAUjOOmZxKJiFxkeDdTl/zQ 6XP+CayFVTRhOq2qTiI5ZfBePhSZV7psBs/DK1i4wWTayQG4ODbf5xBkXlQa201/7FH9y5 cCknCbX4soTwswiSC6AHez/Tk7XhXxGtTy7Z4cniCCuKN139sFp58TyBcNHB5Q== Received: from thor.intern.walstatt.dynvpn.de (dynamic-089-012-099-016.89.12.pool.telefonica.de [89.12.99.16]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id 2EC8C240560; Fri, 6 Dec 2024 21:09:48 +0100 (CET) Date: Fri, 6 Dec 2024 21:09:20 +0100 From: FreeBSD User To: Ronald Klop Cc: freebsd-ipfw@freebsd.org, freebsd-current@freebsd.org Subject: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: Operation timed out Message-ID: <20241206210947.3ae835e4@thor.intern.walstatt.dynvpn.de> In-Reply-To: <279848701.11738.1733510402875@localhost> References: <20241206034709.4dd32cc5@thor.intern.walstatt.dynvpn.de> <279848701.11738.1733510402875@localhost> Organization: walstatt-de.de List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: 51ce02 X-Rspamd-UID: 21f4eb X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE] X-Rspamd-Queue-Id: 4Y4j685hScz564g X-Spamd-Bar: ---- Am Fri, 6 Dec 2024 19:40:02 +0100 (CET) Ronald Klop schrieb: > Might be useful to share your ipfw config. Sorry, my posting must have been disturbing (having in mind a "deny any rule and then disabling the FW ...). Well, the IPFW setup itself is explained quickly - I use almost the vanilla rc.conf-issued IPFW (settings: firewall_type="workstation", firewall_logif="YES", firewall_myservices="22/tcp", firewall_allowservices="any"). The hosts in question have the following kernel configuration, I provide the option tags that might be of interest or, if not, just for the record, as they are not part of GENERIC, see below. Also, I'll provide some sysctl setting performed via /etc/sysctl.conf.local, see below. The configuration and settings have been mostly unchanged over a couple of months for now and did not induce trouble so far. As it deemed fit regarding time and my limited skills, I disabled and enabled piece by piece of the MAC_ and NETGRAPH_ options - without any success so far - my "measurement" is fetching emails via claws-mail (all TLS). claws-mail reports "corrupted/broken stream", does have authetication issues and is de facto unusable - it doesn't refresh IMAP based email fetches and doesn't even quit without a hard kill. Another "indicator" is the time taken to "git pull" of ZFS filesystems: cloning and pulling takes unusual long (/usr/src is UFS/FFS, /usr/ports on a ZFS pool and since the problem occured, it makes a mutual difference). While git pull or clone mutually stuck and claws-mail is endlessly fetching/authenticating emails and never responding back in a usable manner, performing "ipfw disable firewall" makes all of a sudden the system work again as usual and expected. As reported - the problem spreads across all of my CURRENT hosts as I'm going to update them towards a recent CURRENT (they all share similar static kernel configs as described here). Most of the boxes do not show the weird reluctant behaviour when pulling via git, but weren't capable of cloning, bailing out with the timeout reported earlier. I use one CURRENT box as my personal desktop, so no other (server) CURRENT show the Email problem in detail as described. And, for the record: I haven't commented out the "options IPFIREWALL" yet in the kernel config ... Kind regards oh [ KERNEL config different from vanilla GENERIC ] options RATELIMIT options ZFS options TCPHPTS options MROUTING options IPSEC options SCTP options MAC_BSDEXTENDED options MAC_PORTACL options MAC_IPACL options MAC_NTPD #options MAC_DO options NETGRAPH options NETGRAPH_IPFW options NETGRAPH_ETHER options NETGRAPH_EIFACE options NETGRAPH_VLAN #options NETGRAPH_NAT options NETGRAPH_DEVICE #options NETGRAPH_PPPOE options NETGRAPH_SOCKET options NETGRAPH_KSOCKET options NETGRAPH_NETFLOW #options NETGRAPH_CAR # IPFW firewall options IPFIREWALL options IPFIREWALL_VERBOSE options DUMMYNET # traffic shaper options BPF_JITTER # adds support for BPF just-in-time compiler. # Pseudo devices not in GENERIC. device enc # IPsec device device stf # 6to4 IPv6 over IPv4 encapsulation device carp # Common address redundancy protocol device lagg # Link aggregation device gre # GRE Tunnel device epair # A pair of virtual back-to-back connected Ethernet interfaces device if_bridge # bridge device device vxlan # Virtual eXtensible LAN interface For the MAC_ Modules: the appropriate OIDs (sysctl) are disabled as far as the MAC module influence the initial behaviour if unconfigured, for instance (/etc/sysctl.conf.local) [ /etc/sysctl.conf.local ] security.mac.bsdextended.enabled=0 security.mac.mls.enabled=0 security.mac.portacl.enabled=0 security.mac.do.enabled=0 security.mac.ipacl.ipv6=0 security.mac.ipacl.ipv4=0 # net.bpf.optimize_writers=1 # net.inet.ip.fw.verbose=1 #net.inet.ip.fw.verbose_limit=10 net.inet.ip.fw.dyn_keep_states=1 > > Van: FreeBSD User > Datum: 6 december 2024 03:47 > Aan: freebsd-current@freebsd.org, freebsd-ipfw@freebsd.org > Onderwerp: Re: HELP! fetch: stuck forever OR error: RPC failed: curl 56 recv failure: > Operation timed out > > > > > > > Am Thu, 5 Dec 2024 17:33:54 +0100 > > FreeBSD User schrieb: > > > > I found the culprit! > > > > Disabling IPFW ("ipfw disable firewall") turns system back to normal! > > > > For the record: on recent CURRENT, since approx. Nov. 30 and/or December 1st CURRENT seems > > to corrupt network connections. > > > > IPFW is compiled statically into the kernel. > > > > The problem sketched below can be reproduced in a more or less obvious manner on recent > > CURRENT: git pull/git clone of a regular FreeBSD source repo or ports via git+https takes > > either a couple of time (up to several mintes to initiate the pull) - or, in some worse > > cases here, the box runs into > > error: RPC failed; curl 56 Recv failure: Operation timed out > > > > claws-mail complains about "corrupted/broken stream", fetching emails takes Aeons - > > forever, the client does not come back even after several hours. > > > > > On Thu, 5 Dec 2024 16:55:00 +0100 > > > Daniel Tameling wrote: > > > > > > > On Thu, Dec 05, 2024 at 11:51:03AM +0100, FreeBSD User wrote: > > > > > On Wed, 04 Dec 2024 17:20:39 +0000 > > > > > "Dave Cottlehuber" wrote: > > > > > > > > > > Thank you very much for responding! > > > > > > > > > > > On Tue, 3 Dec 2024, at 19:46, FreeBSD User wrote: > > > > > > > On most recent CURRENT (on some boxes of ours, not all) fetch/git seem > > > > > > > to be stuck > > > > > > > forever fetching tarballs from ports, fetching Emails via claws-mail > > > > > > > (TLS), opening > > > > > > > websites via librewolf and firefox or pulling repositories via git. > > > > > > > > > > > > > > CURRENT: FreeBSD 15.0-CURRENT #1 main-n273978-b5a8abe9502e: Mon Dec 2 > > > > > > > 23:11:07 CET 2024 > > > > > > > amd64 > > > > > > > > > > > > > > When performing "git pull" und /usr/ports, I received after roughly 5-7 minutes: > > > > > > > > > > > > > > error: RPC failed: curl 56 recv failure: Operation timed out > > > > > > > > > > > > Generally it would be worth seeing if the HTTP(S) layers are doing the right thing > > > > > > or not, and then working down from there, to tcpdump / wireshark and then if > > > > > > necessary into kernel itself. > > > > > > > > > > My skills are limited, according to packet analysis utilizing tcpdum/wireshark (and > > > > > theory,of course). I tried due to "a feeling" my used older Intel based NIC could > > > > > have some checksum issues like in the past (I saw e1000 driver updates recently > > > > > flowing into FreeBSD CURRENT). > > > > > > > > > > > > If fetch fails reliably in ports distfile fetching, then isolate a suitable > > > > > > tarball, and try it again in curl, with tcpdump already prepared to capture > > > > > > traffic to the remote host. > > > > > > > > > > > > tcpdump -w /tmp/curl.pcap -i ... host ... > > > > > > > > > > > > env SSLKEYLOGFILE=/tmp/ssl.keys curl -vsSLo /dev/null --trace > > > > > > /tmp/curl.log https://what.ev/er > > > > > > > > > > > > I would guess that between the two something useful should pop up. > > > > > > > > > > > > I like opening the pcap in wireshark, it often has angry red and black highlighted > > > > > > lines already giving me a hint. > > > > > > > > > > > > The SSLKEYLOGFILE can be imported into wireshark, and allows decrypting the TLS > > > > > > traffic as well in case there are issues further in. Very handy, > > > > > > see https://everything.curl.dev/usingcurl/tls/sslkeylogfile.html for how to do that. > > > > > > > > > > > > If your issues only occur with git pull, its also curl inside and supports similar > > > > > > debugging. Ferreting > > > > > > through https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems/56094711#56094711 should get you similar info. > > > > > > > > > > > > A+ > > > > > > Dave > > > > > > > > > > > > > > > > Thanks for the hints and precious tips! I'll digg deeper into the matter. > > > > > > > > > > In the meanwhile, I updated some other machines running CURRENT since approx. two > > > > > weeks with an older CURRENT to the most recent one - and face similar but not > > > > > identical problems! > > > > > Updating exiting FreeBSD repositories, like src.git and ports.git, show no problems > > > > > except they take longer to accomplish than expected. > > > > > Cloning a repo is impossible, after 10 or 15 minutes I receive a timeout. > > > > > > > > > > On aCURRENT recently updated and worked flawlessly before (CURRENT now: FreeBSD > > > > > 15.0-CURRENT #5 main-n274014-b2bde8a6d39: Wed Dec 4 22:22:22 CET 2024 amd64), > > > > > cloning attempts for 14.2-RELENG ends up in this mess: > > > > > > > > > > # git clone --branch releng/14.2 https://git.freebsd.org/src.git 14.2-RELENG/src/ > > > > > Cloning into '14.2-RELENG/src'... > > > > > error: RPC failed; curl 56 Recv failure: Operation timed out > > > > > fatal: expected 'packfile' > > > > > > > > > > This is nasty. The host now in question has an i350 based dual-port NIC - the host's > > > > > kernel is very similar to the box I reported the issue first time, both do have > > > > > customized kernels (in most cases, I compile several modules like ZFS and > > > > > several NETGRAPH modules statically into the kernel - a habit inherited from a small > > > > > FBSD project I configured (I wouldn't say developed) which does not allow loadable > > > > > kernel modules due to regulations. > > > > > > > > > > I hoped others would stumble over this tripwire in recent CURRENT sources, since the > > > > > phenomena and its distribution over a bunch of CURRENT boxes with different OS states > > > > > seemingly show different behviour. > > > > > > > > > > And for the record: I also build my ports via poudriere and mostly via make. I also > > > > > rebuilt in a two day's marathon all packages via "make -f" - for librewolf, curl and > > > > > so on to ensure having latest sources/packages. > > > > > > > > > > (I repeat myself here again, sorry, its for the record). > > > > > > > > > > Will report in on further development and "investigations" > > > > > > > > > > Kind regards and thanks, > > > > > > > > > > oh > > > > > > > > > > > > > > > > > > This is a shot into the dark but is this a virtual machine? VirtualBox 7.1.0 had some > > > > networking issues that got fixed later. > > > > > > No, pure Hardware and FreeBSD ... > > > > > > > > > > > Otherwise I would start with ping and traceroute to figure out if they show this issue > > > > and where it occurs. > > > > > > > > > > > > > > > > > > -- > > O. Hartmann > > > > > > > > > > -- O. Hartmann