From nobody Fri Aug 29 09:08:23 2025 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cCsrH4QJKz66NfJ for ; Fri, 29 Aug 2025 09:08:31 +0000 (UTC) (envelope-from devgs@ukr.net) Received: from frv21.fwdcdn.com (frv21.fwdcdn.com [212.42.77.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.ukr.net", Issuer "Thawte TLS RSA CA G1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cCsrG5KC0z49f5 for ; Fri, 29 Aug 2025 09:08:30 +0000 (UTC) (envelope-from devgs@ukr.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ukr.net header.s=ffe header.b=QRtnYDl7; dmarc=pass (policy=none) header.from=ukr.net; spf=pass (mx1.freebsd.org: domain of devgs@ukr.net designates 212.42.77.21 as permitted sender) smtp.mailfrom=devgs@ukr.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; q=dns/txt; d=ukr.net; s=ffe; h=Content-Type:MIME-Version:Message-Id:To:Subject:From:Date :Sender:Reply-To:Cc:Content-Transfer-Encoding:Content-ID :Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References :List-Id:List-Help:List-Unsubscribe:List-Subscribe:List-Post :List-Owner:List-Archive; bh=Hn4Aa/MoPM5DAn1250XHQCApZhF2iqnk9Mz54cWmnrg=; b=QRtnYDl7eELKJ2ToUc4r5l6nPVrbxqsH4J1um34Km3XrtHWlHgWAum3A8roOu7 2rqXSNqgyF43uOHQtcwEjG4iN6w0bLZqNMevOtcBU/Xy3lsfy1nFqLqHmBlir6fJ UzaH6HRivSD/TeYX476NDihPQobie9wTftt9bNdp3UE+M=; Received: from [10.10.15.21] (helo=mpop-test1.fwdcdn.com) by frv21.fwdcdn.com with ESMTP ID TZfiPp-UyvyaUhjqX for freebsd-net@freebsd.org; Fri, 29 Aug 2025 12:08:23 +0300 Received: from [10.10.10.90] (helo=-) by mpop-test1.fwdcdn.com with local ID wJjrvZ-98jA9KaOwV for freebsd-net@freebsd.org; Fri, 29 Aug 2025 12:08:23 +0300 Date: Fri, 29 Aug 2025 12:08:23 +0300 From: Paul To: freebsd-net@freebsd.org Received: from devgs@ukr.net by test1.fwdcdn.com; Fri, 29 Aug 2025 12:08:23 +0300 In-Reply-To: <1753769100.0108837000.0kt30ud9@test1.fwdcdn.com> References: <1753769100.0108837000.0kt30ud9@test1.fwdcdn.com> Subject: Re: Kernel deadlocks on 14.3-STABLE with 100GbE card X-Reply-Action: reply Message-Id: <1756457078.0404318000.u4ltbc3f@test1.fwdcdn.com> X-Mailer: mail.ukr.net 5.0 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-Spamd-Bar: ---- X-Spamd-Result: default: False [-5.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[ukr.net:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[ukr.net,none]; R_SPF_ALLOW(-0.20)[+ip4:212.42.77.0/24]; R_DKIM_ALLOW(-0.20)[ukr.net:s=ffe]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_FROM(0.00)[ukr.net]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:8856, ipnet:212.42.77.0/24, country:UA]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[ukr.net]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[ukr.net:+] X-Rspamd-Queue-Id: 4cCsrG5KC0z49f5 Hi! We have finally managed to reproduce this issue with the help of iperf3. We have triggered a kernel panic with `sysctl debug.kdb.panic=1` to collect core dump, when iperf3 process has entered the inf loop. Here is the basic analysis, please ask for more if required: (kgdb) bt #0 cpustop_handler () at /usr/src/sys/x86/x86/mp_x86.c:1530 #1 0xffffffff808deec8 in ipi_nmi_handler () at /usr/src/sys/x86/x86/mp_x86.c:1487 #2 0xffffffff8090c7af in trap (frame=0xfffffe03edeb8f30) at /usr/src/sys/amd64/amd64/trap.c:248 #3 #4 0xffffffff80640e30 in sbcut_internal (sb=sb@entry=0xfffff801b0ec6e00, len=-2145162648) at /usr/src/sys/kern/uipc_sockbuf.c:1585 #5 0xffffffff80640d78 in sbflush_internal (sb=) at /usr/src/sys/kern/uipc_sockbuf.c:1547 #6 sbflush_locked (sb=) at /usr/src/sys/kern/uipc_sockbuf.c:1559 #7 sbflush (sb=sb@entry=0xfffff801b0ec6e00) at /usr/src/sys/kern/uipc_sockbuf.c:1567 #8 0xffffffff807488f3 in tcp_disconnect (tp=0xfffff8034a572a80) at /usr/src/sys/netinet/tcp_usrreq.c:2702 #9 0xffffffff80743897 in tcp_usr_disconnect (so=) at /usr/src/sys/netinet/tcp_usrreq.c:704 #10 0xffffffff80643655 in sodisconnect (so=0xfffff801b0ec6c00) at /usr/src/sys/kern/uipc_socket.c:2085 #11 soclose (so=0xfffff801b0ec6c00) at /usr/src/sys/kern/uipc_socket.c:1920 #12 0xffffffff8053e921 in fo_close (fp=0xfffff801b0ec6e00, fp@entry=0xfffff801a51ab410, td=0x80236a68, td@entry=0xfffff801a51ab410) at /usr/src/sys/sys/file.h:397 #13 _fdrop (fp=0xfffff801b0ec6e00, fp@entry=0xfffff801a51ab410, td=0x80236a68, td@entry=0xfffff80276bcd740) at /usr/src/sys/kern/kern_descrip.c:3756 #14 0xffffffff80541aca in closef (fp=0xfffff801a51ab410, td=0xfffff80276bcd740) at /usr/src/sys/kern/kern_descrip.c:2851 #15 0xffffffff80545e08 in closefp_impl (fdp=, fd=, fp=, td=, audit=) at /usr/src/sys/kern/kern_descrip.c:1324 #16 0xffffffff8090de97 in syscallenter (td=0xfffff80276bcd740) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:193 #17 amd64_syscall (td=0xfffff80276bcd740, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1241 #18 #19 0x000000082510c87a in ?? () Backtrace stopped: Cannot access memory at address 0x820dd0058 (kgdb) fr 4 #4 0xffffffff80640e30 in sbcut_internal (sb=sb@entry=0xfffff801b0ec6e00, len=-2145162648) at /usr/src/sys/kern/uipc_sockbuf.c:1585 1585 next = (m = sb->sb_mb) ? m->m_nextpkt : 0; (kgdb) p len $33 = -2145162648 (kgdb) set $total=(unsigned int)0 (kgdb) set $count=(unsigned int)0 (kgdb) set $next=(struct mbuf*)sb->sb_mb (kgdb) while ($next != 0) >set $total=$total+$next.m_len >set $count=$count+1 >set $next=$next.m_next >end (kgdb) p $total $34 = 2149804648 (kgdb) p (int)$total $35 = -2145162648 (kgdb) p $count $36 = 1484679 As mentioned before, the problem occurs when the socket is being closed. Now we know why. Because of a cast here: m_freem(sbcut_internal(sb, (int)sb->sb_ccc)); When `sb->sb_ccc` grows above the max unsigned value that can be stored in `int` this cast leads to an infinite loop, within this function. As `len` smaller than 0 is basically equivalent to 0 in `sbcut_internal()`. But that's just a part of a problem. Why does the buffer grow this large? Our limit is: kern.ipc.maxsockbuf=157286400 Is it expected to grow so far beyond this limit? The way we managed to reproduce the issue is to simply spam one host with a traffic from another host: Client: iperf3 --parallel 8 --time 10 --bidir --client Server (where bug occurs): iperf3 --server My guess is the limit is not applied on packet basis. But instead, at some other trigger points. And when there is a burst we manage to accumulate so many packets that their total size becomes > 2147483647. The fact that this is a 100GbE card makes it much more likely. > Hi! > It has been a 4th time now that our server had to be hard re-booted. Last two of them in the span of two hours. > It was only a week since the server was in production. > > > ... >