From owner-freebsd-fs@FreeBSD.ORG Wed Nov 21 09:02:12 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CC995C6D for ; Wed, 21 Nov 2012 09:02:12 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1C10C8FC0C for ; Wed, 21 Nov 2012 09:02:11 +0000 (UTC) Received: by mail-ee0-f54.google.com with SMTP id c13so4858370eek.13 for ; Wed, 21 Nov 2012 01:02:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:date:subject:to :message-id:mime-version:x-mailer; bh=YDILIxHQqKN1pro1BH4brkMSznOVnLZnc5tDtmNKRbg=; b=l8VD9B0qYp4gDtS7VZEMLBp1k9R1IrzuYxVVwtes5APXJ7HFTsoQV8DJcpoDTrxp1x SavQo1TlpcPbTRGi09DELDC/frH2+VNIZFNDfvgxwxVvdOwMQgiwBhJ0trgvcvCijjAD PCyLdjFjkB9eKlBWRfZDAOteCMzv1D2JLwNconR4S20xJazYv59B2k2NPfxszgx0KJtR r96NpyRftfL+xFvnUD8PLJOPbBAx4/Jc/3PrTxIzWlXJp3Zi+DN/hDMI/OvadyyJpUC7 TaFiKa02HOOK/FP/5ba/QTaFJP+XZRVgDZgLZvFRk3RCvXyjWzr7Z7E6Rz6EQD3T3R4M xfVg== Received: by 10.14.203.132 with SMTP id f4mr43303107eeo.11.1353488531000; Wed, 21 Nov 2012 01:02:11 -0800 (PST) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id v47sm10190021eeo.9.2012.11.21.01.02.09 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 21 Nov 2012 01:02:10 -0800 (PST) From: Nikolay Denev Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Wed, 21 Nov 2012 11:02:09 +0200 Subject: nfsd hang in sosend_generic To: "freebsd-fs@freebsd.org" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) X-Mailer: Apple Mail (2.1499) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 09:02:12 -0000 Hello, First of all, I'm not sure if this is actually nfsd issue and not = network stack issue. I've just had nfsd hang in unkillable state while doing some IO from = Linux host running Oracle DB using Oracle's Direct NFS. I was watching from some time how the Direct NFS client loads the NFS = server differently, i.e.: with the linux kernel NFS client I see single TCP session to port 2049 = and all traffic goes there, while the Direct NFS client is much more aggressive and creates multiple TCP sessions, and often was = able to generate pretty big Send/Recv-Q's on FreeBSD's side. I'm mentioning this as probably is related. Here's the procstat -kk of the hanged nfsd process : PID TID COMM TDNAME KSTACK = =20 1221 100550 nfsd nfsd: master mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_run+0x8f nfsrvd_nfsd+0x193 nfssvc_nfsd+0x9b sys_nfssvc+0x90 = amd64_syscall+0x5ea Xfast_syscall+0xf7=20 1221 101286 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sleep+0x2ad = sosend_generic+0x25f svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101287 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101288 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101317 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101318 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101319 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101320 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101321 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101322 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101323 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101324 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101325 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101326 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101327 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101328 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101329 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101330 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101331 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101332 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101333 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101334 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101335 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101336 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101337 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101338 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101339 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101340 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101341 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101342 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101343 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101344 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101345 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101346 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101347 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101348 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101349 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101350 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101351 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101352 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101353 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101354 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101355 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101356 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101357 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101358 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101359 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101360 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101361 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101362 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101363 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101364 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101365 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101366 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101367 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101368 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101369 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101370 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101371 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101372 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101373 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101374 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101375 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101376 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101377 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101378 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101379 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101380 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101381 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101382 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101383 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101384 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101385 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101386 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101387 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101388 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101389 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101390 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101391 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101392 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101393 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101394 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101395 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101396 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101397 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101398 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101399 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101400 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101401 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101402 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101403 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101404 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101405 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101406 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101407 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101408 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101409 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101410 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101411 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101412 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101413 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101414 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101415 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101416 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101417 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101418 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101419 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101420 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101421 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101422 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101423 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101424 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101425 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101426 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101427 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101428 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101429 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101430 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101431 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101432 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101433 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101434 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101435 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101436 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101437 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101438 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 1221 101439 nfsd nfsd: service mi_switch+0x194 = sleepq_catch_signals+0x343 sleepq_wait_sig+0xc _sx_xlock_hard+0x299 = sosend_generic+0x107 svc_vc_reply+0x16f svc_sendreply_common+0xaa = svc_sendreply_mbuf+0x59 nfssvc_program+0x219 svc_run_internal+0x684 = svc_thread_start+0xb fork_exit+0x11f fork_trampoline+0xe=20 Here is a netstat output for the nfs sessions from FreeBSD server side: Proto Recv-Q Send-Q Local Address Foreign Address = (state) tcp4 0 37215456 10.101.0.1.2049 10.101.0.2.42856 = ESTABLISHED tcp4 0 14561020 10.101.0.1.2049 10.101.0.2.62854 = FIN_WAIT_1 tcp4 0 3068132 10.100.0.1.2049 10.100.0.2.9712 = FIN_WAIT_1 Linux host sees this : tcp 1 0 10.101.0.2:9270 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 477940 0 10.100.0.2:9712 10.100.0.1:2049 = ESTABLISHED=20 tcp 1 0 10.101.0.2:10588 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:12254 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:12438 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:17583 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:20285 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:20678 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:22892 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:28850 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:33851 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 165 0 10.100.0.2:34190 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:35643 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:39498 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:39724 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:40742 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:41674 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:42942 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:42956 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 477976 0 10.101.0.2:42856 10.101.0.1:2049 = ESTABLISHED=20 tcp 1 0 10.100.0.2:42045 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:42048 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:43063 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:44771 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:49568 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:50813 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:51418 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:54507 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:57201 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:58553 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:59638 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.100.0.2:62289 10.100.0.1:2049 = CLOSE_WAIT =20 tcp 1 0 10.101.0.2:61848 10.101.0.1:2049 = CLOSE_WAIT =20 tcp 476952 0 10.101.0.2:62854 10.101.0.1:2049 = ESTABLISHED=20 Then I used "tcpdrop" on FreeBSD's side to drop the sessions, the nfsd = was able to die and be restarted. During the "hanged" period, all NFS mounts from the Linux host were = inaccessible, and IO hanged. The nfsd is running with drc2/drc3 and lkshared patches from Rick = Macklem.