Date: Mon, 20 Apr 2026 10:11:21 -0400 From: Olgun Adak <olgun.adak@trexquant.com> To: freebsd-stable@freebsd.org Cc: freebsd-net@freebsd.org Subject: [REGRESSION] nfsd TCP socket lockup on 14.3-RELEASE-p9/p10 - Confirmed on Multiple Systems Message-ID: <CAFww=iytpc%2BJJuiPJtTrKKkqZaP%2BZ9OzHnTCzLk=MSEkzGwNzA@mail.gmail.com>
index | next in thread | raw e-mail
[-- Attachment #1 --] Hello FreeBSD Community, We’ve run into a consistent nfsd lockup after moving from 14.3-RELEASE-p8 to p10. We have verified this across two identical bare-metal systems. Reverting to p8 via bectl immediately restores stability on both systems, so this appears to be a regression introduced in the p9/p10 cycle. *The symptoms:* Under NFSv3 load, the nfsd service hangs and becomes unresponsive to all clients. Looking at procstat -kk, we see a deadlock pattern where threads are stuck waiting on soiolock: _sx_xlock_hard -> soiolock -> sosend_generic -> sosend -> svc_vc_reply Several threads are blocked in _sx_xlock_hard while others sit in sbwait. *The environment:* The systems are bare-metal with 2 x dual-port Mellanox ConnectX-6 100GbE (mlx5en) cards. We see the issue regardless of MTU (1500 and 9000). Offloads: - TSO: Enabled - LRO: The issue persists regardless of LRO state (tested with LRO disabled and with software-only LRO). Hardware LRO is disabled in all cases. Relevant tunables: kern.ipc.soacceptqueue=1000 kern.ipc.somaxconn=2000 kern.ipc.maxsockbuf=67108864 net.inet.tcp.sendbuf_max=67108864 net.inet.tcp.sendspace=16777216 net.inet.tcp.sendbuf_inc=262144 net.inet.tcp.recvbuf_max=67108864 net.inet.tcp.recvspace=16777216 vfs.nfsd.srvmaxio=1048576 We have kept the p10 Boot Environments intact and can boot back into them to run any additional debug commands or test patches if someone can help point us in the right direction. Best regards, -Olgun Adak -- This message is intended only for the use of the individual or entity to which it is addressed, and may contain private and confidential information. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Trexquant Investment LP. All information is subject to change without notice. [-- Attachment #2 --] <div dir="ltr"><div><div><p>Hello FreeBSD Community,</p><p>We’ve run into a consistent nfsd lockup after moving from 14.3-RELEASE-p8 to p10. We have verified this across two identical bare-metal systems. Reverting to p8 via bectl immediately restores stability on both systems, so this appears to be a regression introduced in the p9/p10 cycle.</p><p><b>The symptoms:</b> </p><p>Under NFSv3 load, the nfsd service hangs and becomes unresponsive to all clients. Looking at <code>procstat -kk</code>, we see a deadlock pattern where threads are stuck waiting on <code>soiolock</code>:</p><p><code>_sx_xlock_hard -> soiolock -> sosend_generic -> sosend -> svc_vc_reply</code></p><p>Several threads are blocked in <code>_sx_xlock_hard</code> while others sit in <code>sbwait</code>.</p><p><b>The environment:</b> </p><p>The systems are bare-metal with 2 x dual-port Mellanox ConnectX-6 100GbE (mlx5en) cards. We see the issue regardless of MTU (1500 and 9000).</p><p>Offloads:</p><ul><li style="margin-left:15px"><p>TSO: Enabled</p></li><li style="margin-left:15px"><p>LRO: The issue persists regardless of LRO state (tested with LRO disabled and with software-only LRO). Hardware LRO is disabled in all cases.</p></li></ul><p>Relevant tunables:</p><p></p></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><pre style="text-wrap-mode: wrap;"><code role="text">kern.ipc.soacceptqueue=1000<br></code>kern.ipc.somaxconn=2000<br>kern.ipc.maxsockbuf=67108864<br>net.inet.tcp.sendbuf_max=67108864<br>net.inet.tcp.sendspace=16777216<br>net.inet.tcp.sendbuf_inc=262144<br>net.inet.tcp.recvbuf_max=67108864<br>net.inet.tcp.recvspace=16777216<br>vfs.nfsd.srvmaxio=1048576</pre></div></div></div></div></blockquote><div><p></p><p></p><p></p><p>We have kept the p10 Boot Environments intact and can boot back into them to run any additional debug commands or test patches if someone can help point us in the right direction.</p><p></p><p>Best regards,</p></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr">-Olgun Adak</div></div></div></div></div></div> <br> <div><hr></div><font face="Arial, Helvetica, sans-serif"><span style="font-size:13px"><div><font face="Arial, Helvetica, sans-serif"><span style="font-size:13px"><br></span></font></div>This message is intended only for the use of the individual or entity to which it is addressed, and may contain private and confidential information. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Trexquant Investment LP. All information is subject to change without notice.</span></font>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFww=iytpc%2BJJuiPJtTrKKkqZaP%2BZ9OzHnTCzLk=MSEkzGwNzA>