Date: Mon, 20 Apr 2026 10:34:22 -0400 From: Olgun Adak <olgun.adak@trexquant.com> To: Mark Johnston <markj@freebsd.org> Cc: freebsd-stable@freebsd.org, freebsd-net@freebsd.org Subject: Re: [REGRESSION] nfsd TCP socket lockup on 14.3-RELEASE-p9/p10 - Confirmed on Multiple Systems Message-ID: <CAFww=iya1PkGDXeP5PVx_NviwQNx%2Bh4Yuuv4BZXLtfR8ryX_2A@mail.gmail.com> In-Reply-To: <aeY3j_IKV2eUpD3l@nuc> References: <CAFww=iytpc%2BJJuiPJtTrKKkqZaP%2BZ9OzHnTCzLk=MSEkzGwNzA@mail.gmail.com> <aeY3j_IKV2eUpD3l@nuc>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] Hi Mark, I have a truncated output of procstat -kk without -a. PID TID COMM TDNAME KSTACK 2800 105648 nfsd nfsd: master mi_switch _sleep svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common 2800 105679 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105682 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105683 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105685 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105687 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105688 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105701 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105702 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105703 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105704 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105705 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105706 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105707 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105708 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105709 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105710 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105711 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105712 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105717 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105725 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105726 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105727 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105728 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105729 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105730 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105731 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105732 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105733 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105734 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105735 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105736 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105737 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105738 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105739 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105740 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105741 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105742 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105743 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105744 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105745 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105746 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105747 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline 2800 105748 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline Best, -Olgun On Mon, Apr 20, 2026 at 10:26 AM Mark Johnston <markj@freebsd.org> wrote: > On Mon, Apr 20, 2026 at 10:11:21AM -0400, Olgun Adak wrote: > > Hello FreeBSD Community, > > > > We’ve run into a consistent nfsd lockup after moving from 14.3-RELEASE-p8 > > to p10. We have verified this across two identical bare-metal systems. > > Reverting to p8 via bectl immediately restores stability on both systems, > > so this appears to be a regression introduced in the p9/p10 cycle. > > > > *The symptoms:* > > > > Under NFSv3 load, the nfsd service hangs and becomes unresponsive to all > > clients. Looking at procstat -kk, we see a deadlock pattern where threads > > are stuck waiting on soiolock: > > > > _sx_xlock_hard -> soiolock -> sosend_generic -> sosend -> svc_vc_reply > > Would you be able to share full "procstat -kka" output from an affected > system? > > > > > Several threads are blocked in _sx_xlock_hard while others sit in sbwait. > > > > *The environment:* > > > > The systems are bare-metal with 2 x dual-port Mellanox ConnectX-6 100GbE > > (mlx5en) cards. We see the issue regardless of MTU (1500 and 9000). > > > > Offloads: > > > > - > > > > TSO: Enabled > > - > > > > LRO: The issue persists regardless of LRO state (tested with LRO > > disabled and with software-only LRO). Hardware LRO is disabled in all > cases. > > > > Relevant tunables: > > > > kern.ipc.soacceptqueue=1000 > > kern.ipc.somaxconn=2000 > > kern.ipc.maxsockbuf=67108864 > > net.inet.tcp.sendbuf_max=67108864 > > net.inet.tcp.sendspace=16777216 > > net.inet.tcp.sendbuf_inc=262144 > > net.inet.tcp.recvbuf_max=67108864 > > net.inet.tcp.recvspace=16777216 > > vfs.nfsd.srvmaxio=1048576 > > > > We have kept the p10 Boot Environments intact and can boot back into them > > to run any additional debug commands or test patches if someone can help > > point us in the right direction. > > > > Best regards, > > -Olgun Adak > -- This message is intended only for the use of the individual or entity to which it is addressed, and may contain private and confidential information. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Trexquant Investment LP. All information is subject to change without notice. [-- Attachment #2 --] <div dir="ltr"><div>Hi Mark,</div><div><br></div><div>I have a truncated output of procstat -kk without -a.</div><div><br></div><div><font face="monospace">PID TID COMM TDNAME KSTACK</font><br><br><font face="monospace"> 2800 105648 nfsd nfsd: master mi_switch _sleep svc_run nfsrvd_nfsd nfssvc_nfsd sys_nfssvc amd64_syscall fast_syscall_common <br><br> 2800 105679 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105682 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105683 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105685 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105687 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105688 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105701 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105702 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105703 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105704 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105705 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105706 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105707 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105708 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105709 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105710 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105711 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105712 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105717 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105725 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105726 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105727 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105728 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105729 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105730 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105731 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105732 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105733 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105734 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105735 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105736 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105737 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait sosend_generic_locked sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105738 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105739 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105740 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105741 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105742 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105743 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105744 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105745 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105746 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105747 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline <br><br> 2800 105748 nfsd nfsd: service mi_switch sleepq_catch_signals sleepq_wait_sig _sx_xlock_hard soiolock sosend_generic sosend svc_vc_reply svc_sendreply_common svc_sendreply_mbuf nfssvc_program svc_run_internal svc_thread_start fork_exit fork_trampoline </font></div><div><font face="monospace"><br></font></div><div><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div>Best,</div><div dir="ltr"><br></div><div dir="ltr">-Olgun</div></div></div></div></div><br></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Mon, Apr 20, 2026 at 10:26 AM Mark Johnston <<a href="mailto:markj@freebsd.org">markj@freebsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Apr 20, 2026 at 10:11:21AM -0400, Olgun Adak wrote:<br> > Hello FreeBSD Community,<br> > <br> > We’ve run into a consistent nfsd lockup after moving from 14.3-RELEASE-p8<br> > to p10. We have verified this across two identical bare-metal systems.<br> > Reverting to p8 via bectl immediately restores stability on both systems,<br> > so this appears to be a regression introduced in the p9/p10 cycle.<br> > <br> > *The symptoms:*<br> > <br> > Under NFSv3 load, the nfsd service hangs and becomes unresponsive to all<br> > clients. Looking at procstat -kk, we see a deadlock pattern where threads<br> > are stuck waiting on soiolock:<br> > <br> > _sx_xlock_hard -> soiolock -> sosend_generic -> sosend -> svc_vc_reply<br> <br> Would you be able to share full "procstat -kka" output from an affected<br> system?<br> <br> > <br> > Several threads are blocked in _sx_xlock_hard while others sit in sbwait.<br> > <br> > *The environment:*<br> > <br> > The systems are bare-metal with 2 x dual-port Mellanox ConnectX-6 100GbE<br> > (mlx5en) cards. We see the issue regardless of MTU (1500 and 9000).<br> > <br> > Offloads:<br> > <br> > -<br> > <br> > TSO: Enabled<br> > -<br> > <br> > LRO: The issue persists regardless of LRO state (tested with LRO<br> > disabled and with software-only LRO). Hardware LRO is disabled in all cases.<br> > <br> > Relevant tunables:<br> > <br> > kern.ipc.soacceptqueue=1000<br> > kern.ipc.somaxconn=2000<br> > kern.ipc.maxsockbuf=67108864<br> > net.inet.tcp.sendbuf_max=67108864<br> > net.inet.tcp.sendspace=16777216<br> > net.inet.tcp.sendbuf_inc=262144<br> > net.inet.tcp.recvbuf_max=67108864<br> > net.inet.tcp.recvspace=16777216<br> > vfs.nfsd.srvmaxio=1048576<br> > <br> > We have kept the p10 Boot Environments intact and can boot back into them<br> > to run any additional debug commands or test patches if someone can help<br> > point us in the right direction.<br> > <br> > Best regards,<br> > -Olgun Adak<br> </blockquote></div> <br> <div><hr></div><font face="Arial, Helvetica, sans-serif"><span style="font-size:13px"><div><font face="Arial, Helvetica, sans-serif"><span style="font-size:13px"><br></span></font></div>This message is intended only for the use of the individual or entity to which it is addressed, and may contain private and confidential information. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Trexquant Investment LP. All information is subject to change without notice.</span></font>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFww=iya1PkGDXeP5PVx_NviwQNx%2Bh4Yuuv4BZXLtfR8ryX_2A>