Date: Wed, 21 Nov 2012 11:02:09 +0200 From: Nikolay Denev <ndenev@gmail.com> To: "freebsd-fs@freebsd.org" <freebsd-fs@FreeBSD.ORG> Subject: nfsd hang in sosend_generic Message-ID: <C9654D1E-91EE-4A44-930C-ACD28D77FAFB@gmail.com>
next in thread | raw e-mail | index | archive | help
Hello, First of all, I'm not sure if this is actually nfsd issue and not = network stack issue. I've just had nfsd hang in unkillable state while doing some IO from = Linux host running Oracle DB using Oracle's Direct NFS. I was watching from some time how the Direct NFS client loads the NFS = server differently, i.e.: with the linux kernel NFS client I see single TCP session to port 2049 = and all traffic goes there, while the Direct NFS client is much more aggressive and creates multiple TCP sessions, and often was = able to generate pretty big Send/Recv-Q's on FreeBSD's side. I'm mentioning this as probably is related. Here's the procstat -kk of the hanged nfsd process : PID TID COMM TDNAME KSTACK = =20 1221 100550 nfsd nfsd: master mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_run+0x8f nfsrvd_nfsd+0x193 nfssvc_nfsd+0x9b sys_nfssvc+0x90 = amd64_syscall+0x5ea Xfast_syscall+0xf7=20 1221 101286 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sleep+0x2ad = sosend_generic+0x25f svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101287 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101288 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101317 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101318 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101319 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101320 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101321 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101322 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101323 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101324 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101325 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101326 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101327 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101328 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101329 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101330 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101331 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101332 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101333 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101334 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101335 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101336 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101337 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101338 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101339 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101340 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101341 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101342 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101343 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101344 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101345 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101346 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101347 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101348 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101349 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101350 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101351 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101352 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101353 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101354 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101355 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101356 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101357 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101358 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101359 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101360 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101361 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101362 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101363 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101364 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101365 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101366 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101367 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101368 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101369 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101370 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101371 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101372 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101373 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101374 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101375 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101376 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101377 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101378 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101379 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101380 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101381 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101382 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101383 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101384 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101385 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101386 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101387 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101388 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101389 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101390 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101391 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101392 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101393 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101394 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101395 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101396 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101397 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101398 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101399 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101400 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101401 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101402 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101403 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101404 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101405 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101406 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101407 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101408 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101409 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101410 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101411 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101412 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101413 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101414 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101415 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101416 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101417 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101418 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101419 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101420 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101421 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101422 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101423 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101424 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101425 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101426 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101427 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101428 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101429 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101430 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101431 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101432 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101433 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101434 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101435 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101436 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101437 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101438 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101439 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 Here is a netstat output for the nfs sessions from FreeBSD server side: Proto Recv-Q Send-Q Local Address Foreign Address = (state) tcp4 0 37215456 10.101.0.1.2049 10.101.0.2.42856 = ESTABLISHED tcp4 0 14561020 10.101.0.1.2049 10.101.0.2.62854 = FIN_WAIT_1 tcp4 0 3068132 10.100.0.1.2049 10.100.0.2.9712 = FIN_WAIT_1 Linux host sees this : tcp 1 0 10.101.0.2:9270 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 477940 0 10.100.0.2:9712 10.100.0.1:2049 = ESTABLISHED=20 tcp 1 0 10.101.0.2:10588 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:12254 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:12438 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:17583 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:20285 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:20678 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:22892 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:28850 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:33851 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 165 0 10.100.0.2:34190 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:35643 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:39498 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:39724 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:40742 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:41674 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:42942 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:42956 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 477976 0 10.101.0.2:42856 10.101.0.1:2049 = ESTABLISHED=20 tcp 1 0 10.100.0.2:42045 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:42048 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:43063 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:44771 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:49568 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:50813 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:51418 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:54507 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:57201 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:58553 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:59638 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:62289 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:61848 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 476952 0 10.101.0.2:62854 10.101.0.1:2049 = ESTABLISHED=20 Then I used "tcpdrop" on FreeBSD's side to drop the sessions, the nfsd = was able to die and be restarted. During the "hanged" period, all NFS mounts from the Linux host were = inaccessible, and IO hanged. The nfsd is running with drc2/drc3 and lkshared patches from Rick = Macklem.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C9654D1E-91EE-4A44-930C-ACD28D77FAFB>