From nobody Wed Apr 1 06:53:33 2026 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4flwgK4vcPz6W3gT for ; Wed, 01 Apr 2026 06:53:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4flwgK4DXcz3Zw2 for ; Wed, 01 Apr 2026 06:53:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1775026413; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OglOkkx44k7Oezij6oAKGuAzHAeJS9TvsG2dGzjiBwo=; b=Zh0ttQ+KefvBomq2vO5ezyn5fyowtF5fCHHDxrmVBQh4BXG4OdUHJ6XejebusuWeETthee ArfoXuBmo23vp27kkdLUW3qxDnf9WTWxLgk/aquFccfv2VQJzu8eqLlsjddRa+W916PvDn sTNzpV5+iQyFCjt2nsxDPuxUCCSl2GG8fHmbpMObM4AATEXl20MsrtmTfUbIe8UcxXZW35 VALlp5EUKbq39EfhqkKXeQj4mrCzLQd4Qp0eDxEt7fvUwuPcDUtgVWiS3y4RlYBAWit6+m pcnLag21vnWEOGXbwOXbhw9Wf7yASkSrxcTvQx5jGkVEsSGLQftUPQYaOZXpIw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1775026413; a=rsa-sha256; cv=none; b=FtnZn9nt6mOWlNjJisE2P2lSM3hEEtmqeQ2zNRg2W6cfeA+g9BtDlxSlbth7BqCO4BmT+Z Cva9uEBQIj6RhF/T1LXE2Iald/ic+jWwzj5AnD2i9lstqlpqUQfyxNGgf+llOo6p5KTEHE JGnRRsbDV2hOUH53EXZRB1APruizGx5e1JL6nsdq40CtJ7nR9z74EYY76o2I1qkJlQtW4n i62CxzqeFlfRI6Y4QisQEDk1S/i9X+y6uRy/Y2bG4rrhur/gOg5HlVxVPQtXTg9BZuuVis rFIDSKCUt2h1+thuLCfnP8ESDfOb8NCZFIYVhoXr9vN3ge+fULC9HuUBjP/ztw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1775026413; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OglOkkx44k7Oezij6oAKGuAzHAeJS9TvsG2dGzjiBwo=; b=mJBtQk19hicibdTw3zL9t48KM/N1mLSA6hLmJQF4JeuzY8y33iUVKLrk8/jzExco9vfdQt n5ra39JWrqZPUM9NsT904Be2sgT0FLgG4h0zAkwCu6t26PgwvtJCoKpVuS2BP+ZuJFMPhz NQRKCCn9wRiBc9snmyo0IE9HtX7hWzeD90YzcZnIUESDcnZPOzglip+wTumDA2UDyb+Jx/ v6VDgpbYq8SIGy+8mckznuWBEK1ulr3/yArAgea3cJjlHpZ+tC8Y/XhjrFKDrZVHTqBGaG puQuNmXbb5PfEI0J+WA6djcnYrqen7x3EEdgxLIlzMvEkANMKI8IIPeA0ve4YA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4flwgK3drYz14Qg for ; Wed, 01 Apr 2026 06:53:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 6316rXqT025463 for ; Wed, 1 Apr 2026 06:53:33 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 6316rXsd025462 for bugs@FreeBSD.org; Wed, 1 Apr 2026 06:53:33 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 293382] Dead lock and kernel crash around closefp_impl Date: Wed, 01 Apr 2026 06:53:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.3-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: devgs@ukr.net X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D293382 --- Comment #41 from Paul --- (In reply to Kyle Evans from comment #38) Basically, this is a HTTP/Websocket micro-service app. It handles hundreds = of thousands of long-living incoming connections. Approximately 98% of them are Websocket and the rest is plain HTTP. Other than that, there are also a lot= of outbound HTTP connections, somewhere in range of 200k-300k. Yeah, to be abl= e to implement this we need timers, a lot of them. All of it is based on a standalone asio C++ library. It does the heavy lift= ing under the hood. We use several `io_context` for scale. And I believe each of them has its own kqueue, which seems reasonable. Not sure about EV_DELETE/EV_ONESHOT, as we're not dealing with kqueue direc= tly. Looking at source code I see no EV_ONESHOT and even see that earlier they've abandoned its use (https://www.boost.org/doc/libs/latest/doc/html/boost_asio/history.html#boo= st_asio.history.asio_1_4_9___boost_1_46_1). It does use EV_DELETE, seemingly on every descriptor close, so... a lot. Connections appear and fall off organically. A lot of them on timeout, as t= his is the real world and not all browsers bother to do the clean Websocket shutdown. This service is nothing new for us. We've been using it for years. Even tod= ay, we're still keeping a fallback server that runs FreeBSD 13.1-STABLE but, and this is a big but: with a different server platform : 2x E5-2660. So, not o= nly have we switched to a newer FreeBSD version, but also dramatically changed = the platform. And the current platform is much, much faster. My gut feeling is: there was a problem already, it just became apparent. Because, a faster CPU makes things, both in kernel and userspace, run faster. And some, highly improbable races, become likely. We've been running this platform and OS version for months now, under diffe= rent kinds of load: not that many sockets, but much more CPU usage overall. And = it runs completely stable. On the other hand, for this specific role, we now h= ave 2 (of 2) servers that exhibit these crashes. Which doesn't rule out some CPU issues yes, but makes it highly unlikely to be the cause. --=20 You are receiving this mail because: You are the assignee for the bug.=