From nobody Sat Nov 18 15:00:14 2023 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SXcQ65ncgz51CxN for ; Sat, 18 Nov 2023 15:00:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SXcQ631z1z3DQ5 for ; Sat, 18 Nov 2023 15:00:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1700319614; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=kQV9gMW+BDc0m0cxmu9TqTFJWiOMVpFGEEQOjCCt4HM=; b=GeqBXP6BXoU6itEVklKWwvbs4FzuCD0XKvzTznt44UegBYzRiFYxT1oXytrhA+xuB7LBfo n02X8KggHBekQYFkqv+blyDfsr90nUx4BFg0hO31otqsL/dGFxtCkWOjY6kxbgUmM3NYyT pDiUFHCCcPDExb7Ug5eCYtAKLHlBrrkkW+iOZdLlkN3GTzC0exXagzRi36e3BUL8X3Jj3+ JxaZzyCm8oY0CBsYD24W7Kd4f0/vUBisl0MTLniuZOwCjr4YbDAb9KLcQeV8qUlSzA5ZrW PsdY5Bm/JDL6pW/AnliLD7c3E/smghER/Yaf9ccXgEsrpckazW8ZhjvsAnnI4w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1700319614; a=rsa-sha256; cv=none; b=RisLfv7bJNPPuSE4Nq814ldu4OQjhTzpOrMY39zHnOCHfKDNRZT48qav8kEUZwfxDb+G5q 57gFP0QfHcjr+jq4yk9mMLrTUvLg6ct0OMnnFqsRsSpXnvBR4frUWdA0vrzKLvOhNLQksd zUqyB2sqK/OPqmgCdsRIsA4KEsGfxU4b4nX0XjDUXtAckuVO2T+ZVVPPk2fFAJNEo0cFdH 5dS8abbe/DrhbHhP+CICJZV0qK9gBLXNa6fwlbmjHHeuFPZRJTFllGf1f7p8/jTcCNHYr7 spjsO69Y8JJxdA5/Jlh2nOdyG7q/vd4OZ+Jhgw7LZ7FJsqUjJnF73NlRdlzcWg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SXcQ6260KzX5w for ; Sat, 18 Nov 2023 15:00:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3AIF0EVw004382 for ; Sat, 18 Nov 2023 15:00:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3AIF0EZs004381 for bugs@FreeBSD.org; Sat, 18 Nov 2023 15:00:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 275169] Panic: rw_rlock: wlock already held for tcpinp @ /usr/src/sys/netinet/in_pcb.c:2529 Date: Sat, 18 Nov 2023 15:00:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: i.dani@outlook.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275169 Bug ID: 275169 Summary: Panic: rw_rlock: wlock already held for tcpinp @ /usr/src/sys/netinet/in_pcb.c:2529 Product: Base System Version: 13.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: i.dani@outlook.com Created attachment 246389 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D246389&action= =3Dedit Kernel config After we upgraded some of our hosts from FreeBSD 12.4 to 13.2 we started se= eing some of our physical hosts freezing randomly and without panicing. We were able to narrow it down to some IPFW rules. The setup is the followi= ng: - Host A: Recently upgraded physical host with FreeBSD 13.2 - Host B: Also a physical host with FreeBSD 13.2 which runs a webserver, serving our own pkg-repos (10.1.1.20). Webserver doesn't matter - we tried Apache, nginx, thttpd. Filetype also doesn't matter (.pkg, .txt, whatever := )). We also tried with multiple hosts. Host A has following IPFW rule: ipfw add 1000 allow ip from me to 10.1.1.20/32 uid 0 Host B has the following IPFW rule: ipfw add 2000 allow tcp from any to 10.1.1.20 80,443 keep-state We can reproduce the freeze by repeatedly fetching a file on Host A from Ho= st B (we initially triggered the bug when running "pkg upgrade"): [root@host-a] $ while true; do curl -v http://10.1.1.20/test.txt --output /dev/null; done Sometimes immediately, sometimes after a few seconds the network connection= of Host A is lost. Sometimes we are still able to log in through a local shell. Sometimes after a few seconds, sometimes after 1-2 minutes the host freezes completely. There is no kernel panic and nothing in the logs. Host B is sti= ll running fine and doesn't freeze. Whats interesting: - Freezes do NOT happen if the "uid 0" selector from Host A's rule is remov= ed. - Freezes do NOT happen if the "keep-state" of Host B's rule is removed. - Freezes do NOT happen with our virtual servers - only physical hosts are affected. After building and installing the kernel with debug options, we were finally able to cause a panic and get some more informations. The Kernel has been b= uilt with the following options: makeoptions DEBUG=3D-g options INVARIANTS options INVARIANT_SUPPORT options WITNESS options WITNESS_SKIPSPIN options DIAGNOSTIC You can find the full kernel config attached (config.txt). PANIC: rw_rlock: wlock already held for tcpinp @ /usr/src/sys/netinet/in_pcb.c:2529 Attached you find: - HW-Info.txt: Hardware information of one of the hosts that freezes. Other hosts that freeze (and also Host B of the example abvoe) are physical too a= nd also use the same NIC-Driver (ix). - info.txt: File written @ Panic - Contains FreeBSD version info and so on. - config.txt: Kernel config (see above). - ddb.txt: Contains the ddb-Dump - reduced to the relevant stuff (panic, backtrace, locks). The full ddb-Dump (containing all procs) can be provided= if needed. Any help in further debuging or fixing this would be highly appreciated! --=20 You are receiving this mail because: You are the assignee for the bug.=